EVIDENCE-BASED PROJECT, PART 3: CRITICAL APPRAISAL OF RESEARCH

Starting with level I evidence studies and moving down the hierarchy list, the EBP team takes each study and, one by one, finds and enters its essential elements into the first five columns of the evaluation table (see Table 1 ; to see the entire table with all 15 studies, go to http://links.lww.com/AJN/A11 ). The team discusses each element as they enter it, and tries to determine if it meets the criteria of the critical appraisal guide. These elements-such as purpose of the study, sample size, and major variables-are typical parts of a research report and should be presented in a predictable fashion in every study so that the reader understands what's being reported.

Table 1Opens a popup window

Opens a popup window

As the EBP team continues to review the studies and fill in the evaluation table, they realize that it's taking about 10 to 15 minutes per study to locate and enter the information. This may be because when they look for a description of the sample, for example, it's important that they note how the sample was obtained, how many patients are included, other characteristics of the sample, as well as any diagnoses or illnesses the sample might have that could be important to the study outcome. They discuss with Carlos the likelihood that they'll need a few sessions to enter all the data into the table. Carlos responds that the more studies they do, the less time it will take. He also says that it takes less time to find the information when study reports are clearly written. He adds that usually the important information can be found in the abstract.

Rebecca and Chen ask if it would be all right to take out the "Conceptual Framework" column, since none of the studies they're reviewing have conceptual frameworks (which help guide researchers as to how a study should proceed). Carlos replies that it's helpful to know that a study has no framework underpinning the research and suggests they leave the column in. He says they can further discuss this point later on in the process when they synthesize the studies' findings. As Rebecca and Chen review each study, they enter its citation in a separate reference list so that they won't have to create this list at the end of the process. The reference list will be shared with colleagues and placed at the end of any RRT policy that results from this endeavor.

Carlos spends much of his time answering Rebecca's and Chen's questions concerning how to phrase the information they're entering in the table. He suggests that they keep it simple and consistent. For example, if a study indicated that it was implementing an RRT and hoped to see a change in a certain outcome, the nurses could enter "change in [the outcome] after RRT" as the purpose of the study. For studies examining the effect of an RRT on an outcome, they could say as the purpose, "effect of RRT on [the outcome]." Using the same words to describe the same purpose, even though it may not have been stated exactly that way in the study, can help when they compare studies later on.

Rebecca and Chen find it frustrating that the study data are not always presented in the same way from study to study. They ask Carlos why the authors or journals wouldn't present similar information in a similar manner. Carlos explains that the purpose of publishing these studies may have been to disseminate the findings, not to compare them with other like studies. Rebecca realizes that she enjoys this kind of conversation, in which she and Chen have a voice and can contribute to a deeper understanding of how research impacts practice.

As Rebecca and Chen continue to enter data into the table, they begin to see similarities and differences across studies. They mention this to Carlos, who tells them they've begun the process of synthesis! Both nurses are encouraged by the fact that they're learning this new skill.

The MERIT trial is next in the stack of studies and it's a good trial to use to illustrate this phase of the RCA process. Set in Australia, the MERIT trial 1 examined whether the introduction of an RRT (called a medical emergency team or MET in the study) would reduce the incidence of cardiac arrest, unplanned admissions to the ICU, and death in the hospitals studied. See Table 1 to follow along as the EBP team finds and enters the trial data into the table.

Design/Method. After Rebecca and Chen enter the citation information and note the lack of a conceptual framework, they're ready to fill in the "Design/Method" column. First they enter RCT for randomized controlled trial, which they find in both the study title and introduction. But MERIT is called a "cluster-randomised controlled trial," and cluster is a term they haven't seen before. Carlos explains that it means that hospitals, not individuals or patients, were randomly assigned to the RRT. He says that the likely reason the researchers chose to randomly assign hospitals is that if they had randomly assigned individual patients or units, others in the hospital might have heard about the RRT and potentially influenced the outcome. To randomly assign hospitals (instead of units or patients) to the intervention and comparison groups is a cleaner research design.

To keep the study purposes consistent among the studies in the RCA, the EBP team uses inclusive terminology they developed after they noticed that different trials had different ways of describing the same objectives. Now they write that the purpose of the MERIT trial is to see if an RRT can reduce CR, for cardiopulmonary arrest or code rates, HMR, for hospital-wide mortality rates, and UICUA for unplanned ICU admissions. They use those same terms consistently throughout the evaluation table.

Sample/Setting. A total of 23 hospitals in Australia with an average of 340 beds per hospital is the study sample. Twelve hospitals had an RRT (the intervention group) and 11 hospitals didn't (the control group).

Major Variables Studied. The independent variable is the variable that influences the outcome (in this trial, it's an RRT for six months). The dependent variable is the outcome (in this case, HMR, CR, and UICUA). In this trial, the outcomes didn't include do-not-resuscitate data. The RRT was made up of an attending physician and an ICU or ED nurse.

While the MERIT trial seems to perfectly answer Rebecca's PICOT question, it contains elements that aren't entirely relevant, such as the fact that the researchers collected information on how the RRTs were activated and provided their protocol for calling the RRTs. However, these elements might be helpful to the EBP team later on when they make decisions about implementing an RRT in their hospital. So that they can come back to this information, they place it in the last column, "Appraisal: Worth to Practice."

After reviewing the studies to make sure they've captured the essential elements in the evaluation table, Rebecca and Chen still feel unsure about whether the information is complete. Carlos reminds them that a system-wide practice change-such as the change Rebecca is exploring, that of implementing an RRT in her hospital-requires careful consideration of the evidence and this is only the first step. He cautions them not to worry too much about perfection and to put their efforts into understanding the information in the studies. He reminds them that as they move on to the next steps in the critical appraisal process, and learn even more about the studies and projects, they can refine any data in the table. Rebecca and Chen feel uncomfortable with this uncertainty but decide to trust the process. They continue extracting data and entering it into the table even though they may not completely understand what they're entering at present. They both realize that this will be a learning opportunity and, though the learning curve may be steep at times, they value the outcome of improving patient care enough to continue the work-as long as Carlos is there to help.

In applying these principles for evaluating research studies to your own search for the evidence to answer your PICOT question, remember that this series can't contain all the available information about research methodology. Fortunately, there are many good resources available in books and online. For example, to find out more about sample size, which can affect the likelihood that researchers' results occur by chance (a random finding) rather than that the intervention brought about the expected outcome, search the Web using terms that describe what you want to know. If you type sample size findings by chance in a search engine, you'll find several Web sites that can help you better understand this study essential.

Be sure to join the EBP team in the next installment of the series, "Critical Appraisal of the Evidence: Part II," when Rebecca and Chen will use the MERIT trial to illustrate the next steps in the RCA process, complete the rest of the evaluation table, and dig a little deeper into the studies in order to detect the "keepers."

REFERENCE

1. Hillman K, et al. Introduction of the medical emergency team (MET) system: a cluster-randomised controlled trial. Lancet2005;365(9477):2091-7. [Context Link]