Research Assignment
Transcript: Analyzing Evidence and Drawing Conclusions
Hi everyone, today’s lecturette is titled "Analyzing the Evidence and Drawing Conclusions." And if you will think about all of the steps in the research process that we have gone over so far, this is the one that challenges your brain the hardest.
Slide 1: Analyzing the Evidence and Drawing Conclusions
This is the one where you are serving as the computer to look at all of the evidence that we have, think carefully about it, and draw a conclusion. So this is where you really have to think about being systematic, and careful, and following a process where you try to have as little bias as possible introduced into the process. So I'm going to try and take you through it and give you as many hints as I can about how to analyze and draw a conclusion that anyone else would draw in exactly the same way. And that’s really what we mean when we talk about rigor. In rigor, what that means is that you've been so careful, that even if you would give the project to ten other people to do, they would draw the same conclusion. So what that means is everyone has to be careful with the evidence, and to analyze it effectively. So I'm going to take you through the process of doing that.
Slide 2: Steps in the EBP process
Remember, these are the steps in the evidence based practice process. And today, we're talking about the third step, critically analyzing the evidence. 0:01:43.3
Slide 3: Research synthesis is the way we analyze the evidence
All right, the way we analyze the evidence is to do a process that is called research synthesis. And if you'll recall, synthesis means that you are making something new out of something that exists. And that is really what you are doing when you're talking about evidence based practice, you're taking all of the literature that exists and you're creating something new, and that something new is the best practices. That's what you're aiming for with your synthesis. And so you can see here that the research synthesis is an integration of research according to scientific principles and rules. So that means that you're being very careful with what you do, you're treating every piece of literature in exactly the same way, you're rating the quality of that literature, and then you're determining whether or not there is a preponderance of evidence that a particular intervention is more preferable to another intervention. The goal here then, is to make it so carefully done, as I said earlier, that other scientists could replicate it; and when others can replicate it, then we have more confidence that the conclusions we've drawn are indeed the best representation of reality as we can get at the present time. 0:03:14.9
Slide 4: If life were perfect, research synthesis would give you …
Now here, if life were perfect, at the end of your research synthesis, you would have one of these items. These are our goals, no matter what your topic happens to be, what we are looking for is a protocol, a best practices, a state of the science, a standard of care. What we're looking for is a statement that is definitive enough, and it’s supported by evidence, that we can say, ”All right, there needs to be a change in practice and this is the way we need to be doing things.” So, this is what we are looking for when we are doing that research synthesis.
Slide 5: What is required to do research synthesis?
Now, these are the things that are required to do research synthesis. And I know this might seem a little stupid, but of course, you've got to have that burning question, remember, it’s that PICO definition that came out of your burning question that gives you your parameters for your search. So obviously you have to have that. The other thing you have to have is a body of research. You cannot do evidence-based practice without having the existing literature. And that is somewhat of a challenge for nursing, because as I've said before, we're a very new science. We're just beginning to develop our body of knowledge; and so we have many clinical questions that are asked, you to go to the literature and you find out, there is nothing there. And you simply can't do synthesis, research synthesis, unless you can find research in the literature. So as you go about doing your own project, this probably might be one of the first questions you want to answer, and that is, "Is there a body of evidence even out there? Is there research literature I can get through a search that will allow me to go through this process?" And if the answer to that is no, then you probably have to modify your question or look in a different area, because you have to have a research literature. 0:05:15.4
The next thing that you need is an evidence table, and I'm going to show you what an evidence table looks like and how to use it in the process of doing your research synthesis. And then finally, a logical conclusion: what is it that you can conclude from the literature that you have synthesized. So it's really that conclusion that is the something new that came from others that existed that makes the synthesis.
Slide 6: How do we organize evidence?
All right, how you organize an evidence table. I'm sorry, the purpose of an evidence table is to allow you to be able to see, side-by-side, a variety of evidence. So this is kind of a high-fallutin' way to describe the evidence table, but we build the evidence table to juxtapose, or compare side-to-side - that's what juxtapose means - juxtapose literature elements for easy comparison. So you're going to create a table that contains all of the literature that you find. And so you'll be able to look at that literature, more or less, side by side, and then you can draw conclusions from it. So the evidence table really is a heuristic, as it’s called, it is simply a mechanism to improve the quality of the research synthesis. 0:06:47.0
Slide 7: What goes on an evidence table?
And these are some of the items that could go on an evidence table. But the evidence table should be created specifically for the topic that you are addressing. So you may want or need to have different elements. But if you were doing what is sort of considered the prototypical evidence-based practice project, you would be comparing two interventions. And if that were the case, these are very traditional items that you would have on your evidence table. First of all, you would need to have the citation, and you don't have to have the full citation if you simply want to just have the first author and the year, if you want to number your literature, it doesn't matter, just so you're able to go back to the literature, if you should need to get more from it that you didn't put on your evidence table. The second item is the quality rating, and I'm going to talk about that further on in the lecturette today. But we have quality ratings that go from 1 to 7. And I highly recommend that you put your literature together with like quality ratings together, so that really does help you look at it and draw conclusions from it. The third thing that you want is the setting. Remember, you want that setting to be somewhat close to your PICO definitions. So it's important to know the setting. The participant characteristics and number. Larger numbers mean that we have more confidence in the conclusions drawn from the research; smaller numbers means that we have less confidence. We need more studies with smaller numbers, if you're following me. So, that numbers do really count. You also wan the hypothesis that was being tested in the research study. You also want to know what was the method or the design that they used. Remember, as you review the lecture that you had from 790 on designs and threats to internal and external validity, that design makes a big difference in the confidence that we have about cause and effect. So knowing the design and understanding whether it protects against threats to internal and external validity is an important process of looking at the quality of the evidence that you have. You also want to know the statistical findings, the P values; and the P value is going to tell you how confident we are that the differences that occurred in the research were due to the intervention. 0:09:39.8
Then you want to see, what does the author of the research say in terms of conclusions? When an author submits a manuscript for publication, that conclusions section is his or her opinion, shall we say, about the meaning of the research that was done. So use their opinion, let them help you understand the meaning that comes from that study. And then finally, I always like to leave a column where I can put comments; so it's just notes basically, to yourself, about what you think about the study, or anything extra that might have come out of your reading of the article that is not necessarily in one of your columns. So it just gives you basically a place to write comments should you have any.
Slide 8: An evidence table looks like this:
Now here is what an evidence table looks like. This is perhaps a little strange, because the titles are going up and down as opposed to side by side. You can do yours this way too, and that way it wouldn't be quite such a large table. Generally speaking, when you're doing evidence tables, it’s going to be landscaped because you're going to have multiple, multiple columns so that you can extract the same information from many studies and then look at it all at once. 0:11:08.3
So remember that evidence table is like your working space, where you can take from the research article the specific information that you need, and then compare article to article.
Slide 9: Now that we have an evidence table, we examine the data for quality
Now that you have the evidence table, what you're going to do is look at the quality of the study. And in your text, you'll find that there was a quality indicator. The quality indicator is very important. And part of the process of understanding the quality is to make sure that you have exactly the right articles to be working with. So I want you to ask yourself these questions: Should the article be in included in my analysis? You need to make inclusion and exclusion standards before you ever begin analyzing or synthesizing the evidence that you found. And by inclusion and exclusion criteria, what I mean is, how would you know if an article should or should not be included in your final synthesis. And I'll give you some hints on how you might know that. First of all, how close is the article to your PICO definitions? The more closely matched the article is to your PICO definitions, the more certain you are you need to include it in your synthesis. There isn't a right or wrong answer to this question. It is a subjective decision that you'll need to make; but, once you've gotten into the literature, you'll be pretty familiar with it and I don't think you'll have a whole lot of questions. But one of the criteria you use is how close is it to my PICO definition? The next question you might ask yourself: Is it really comparing the items that I want to compare, the interventions, the assessment tools, the things that we've talked about in the past? You will, perhaps, find some articles that are comparing head-to-head exactly the two interventions you wanted compared. That does not happen all the time. As a matter of fact, unfortunately, that's not the most common thing that you'll find. You'll actually find a lot of research studies that are comparing your interventions of interest, but they'll have other things that they are comparing as well. That's okay, as long as you're able to get the information about your two interventions out of that research study. 0:14:11.7
Commonly, a study will compare three different interventions, and maybe they'll compare two interventions to one, and it’s not the head-to-head comparison that you want. So what you'll have to do is read the study, and decide whether or not the study gives out information about the head-to-head comparison that you wanted. If it does not, then you probably don't want to use it in your synthesis. So just to reiterate, there are two ways that you can tell whether or not you should put the article in your synthesis; and those are: how close is it to PICO definitions, and are you able to really get information about your comparisons from the study. And there may be some other inclusion and exclusion criteria that you'll come up with based upon the study topic of your EBP project. Then, you'll also want to determine whether or not the study is valid. And when you're talking about something being valid, the major idea is that the outcome, the change in outcome, was due to the intervention and nothing else, no other outside variable were causing that change. And there are some ways that we can assure validity, and that has to do with the designs. 0:15:53.8
Slide 10: Quality of Evidence
These are the Quality of Evidence levels, and they go from levels 1 through 7. And these levels are based exclusively on the design and the number of studies that have been done. Those are the two things that they use for validity. And the designs, as you've reviewed, are experimental, quasi-experimental, descriptive, qualitative; we have a large number of different kinds of designs that we could use. And we want the ones that control for threats to internal validity the most successfully. Those are most valued designs. So the whole levels are based upon those designs and the number of studies that have been done. So, in level 1, these are the systematic reviews and meta-analyses. Systematic reviews means that someone else, presumably an expert in the field, has pulled together all of the studies that have been done, in a particular area and drawn a conclusion. In other words, somebody else has done your work for you; because basically, this is what I'm asking you to do, is do a systematic review, and draw conclusions from it. So you may find those. These are especially found in the Cochrane Review, and in JoAnna Briggs, and in some of the other sites that I've given you, that focus on systematic reviews. You also may find these in guidelines; in discussion of guidelines and how panels came up with the guidelines, there may have been systematic reviews. 0:17:48.7
Now the thing that is great about level 1 is that you have multiple studies. So the more studies that you have that state that intervention A has better outcomes than intervention B, the more confident you can be that that indeed is true. Should I use that word “true”? Meta-analysis is very much the same as a systematic review with one exception. In the meta-analysis, there is actually a mathematical formula that is used to calculate the effect size for the multiple articles that were done, comparing two interventions. So, it is just as valuable as a systematic review, plus you have the increased confidence of the effect size that tells you just exactly how strong the difference is between the two interventions. So, level 1 is the highest quality evidence. It’s the level we can have the most confidence in; and it’s because it’s multiple studies testing the same comparison with statistical findings that can be looked at together and conclusions can be drawn. Level 2 is a randomized clinical trial. You'll see that abbreviated RCT. The randomized clinical trial is the gold standard in terms of research design. When you have a randomized clinical trial, what that means is, every subject is randomly assigned to treatment group, and the researcher and the data collector are blind to group. So those are aspects of the randomized clinical trial that really increase the validity and protect against all of the threats to internal validity. So, it’s a high quality study, and the reason that that does not appear at level 1, is because we are talking here about just one study. One randomized clinical trial, especially if you have a large number of subjects, is a very powerful piece of evidence that you can use when you're drawing your conclusion. It would be great if we had multiple randomized clinical trials; that would be our highest level of confidence that one intervention was better than the other. 0:20:40.6
Level 3 is experimental and quasi-experimental research without randomization. When you are dealing with people, with human beings, sometimes it’s not possible to randomize. Or sometimes it is not possible to have the data collector blind to group. These are still high quality designs, but they don't protect against threats to internal validity quite as well as the randomized clinical trial does. So that is why they are considered a level down. And then level 4 is a case controlled cohort study. This is the very common design used in public health. And often what you'll find is cases that are occurring now, and they will be compared to cases that occurred in the past. And the case information has been stored in a very large database. So the researchers are able to compare what has happened in the past with what is happening now. And the cohorts are matched, often by gender, by age, by disease, etc. So you have confidence that you are comparing apples and apples. But the thing that is different here is that you don't have both groups moving forward in time together. You don't have that going on. You have one that has already happened and you're comparing it something that is happening now. So that's what makes us have less confidence in it, because we're not protecting, especially, the threats of history in this particular design. So these are the highest level. If you have a predominance of studies that are in these four levels, then you can feel pretty good. You can say 'My evidence is pretty high quality.' Now, as I said earlier, when you're doing your evidence table, it’s really important that you put levels together. So always start your evidence table with all of the level 1 studies that you have. And then, after you've put all of those in your evidence table, put the level 2s, the level 3s, and the level 4s; keep everything together. It makes it so much easier when you're trying to draw conclusions about what you have. 0:23:25.4
Slide 11: Quality of Evidence
Now, we're moving in to levels of evidence that are not nearly as valid as the others. And, just from talking about designs, I think you'll understand clearly why. Level 5 is evidence from systematic reviews of descriptive and qualitative research. Now you know by definition, that descriptive and qualitative research, you never have randomization. You're not even in comparing things. You're just describing what currently exists. These studies are very important in starting a research trajectory. But they do not at all speak of cause and effect, there are no controls for internal or external validity. So, they have very little power in the way of validity. You have to be extremely careful in drawing conclusions about changing practice based upon a level 4 or level 5 evidence. Level 6 is even a lower level, and this is from evidence from a single descriptive or qualitative study. Here you can see once again, the levels of evidence are based upon the design, how valid it is, and on the number of studies that have been done. More studies is better, the tighter designs are better. Protection against threats to internal validity is very important. 0:24:55.8
And then finally we have level 7, which is the evidence from the opinion of experts or expert panels. And this is a level that, the lowest level, and unfortunately it’s the kind of evidence that we often base our practice on; but we shouldn't because it’s the lowest level of evidence, it is one person's opinion. There is no control for bias; there is no control for threats to internal validity; there is no randomization. It's just simply a person's opinion. Now, when you are doing your synthesis or drawing your conclusion from the evidence that you have, you want to look at the quality, and you also want to look at the numbers that you have in each one of these levels. And just as an example, you could have a hundred level 7s, and two level 1s, and the level 7s would never outweigh the level 1s. I hope you're getting what I mean. The weight and the influence in your decision-making is much more profound when you are talking about level 1s, 2s, and 3s. Even if you've only got three or four in that category and you've got more in levels 5, 6, and 7, the levels 1, 2, and 3 always have the most weight. They are the best studies, they are the most controlled, they contain the most subjects; they carry the most weight when you are trying to draw your conclusion about what the evidence tells us. So that's why if you use your evidence table, put like things together: you can really count subjects, count number of studies, and come up with a way of drawing a conclusion that you feel relatively confident with. 0:27:01.3
Slide 12: How do I make sure that I am not introducing bias?
Now, how do I make sure that I'm not introducing bias? I'm sure that when you studied designs, you realized that protecting against bias in any of its forms is what makes a study valid. So when you're doing a research synthesis, you're also worrying about validity here. You don't want to introduce bias. And even though we might try very hard not to be biased, most of us go into these processes, either in a traditional research study or in an EBP, with an idea about which intervention works the best. Try as we might, that's always in the back of our head, we always want it to come out the way we think it should come out, and that's bias. So we want to try and get rid of that bias. And one of the ways is for you to have multiple people in your group do the analysis independently. You can create one evidence table, and then give everybody a copy of that evidence table, and then have everyone do the research synthesis independently, then come back together and compare notes. And if you have differences of opinion, then talk it through. Figure out what is the right conclusion to come to. So in essence, you're putting up checks and balances on each other. Maybe all of you don't have the same thought about which intervention is the best one. And so, by doing it independently, and comparing notes, talking it through, figuring out what is the most valid conclusion you can come to, that increases your confidence that the conclusion you have drawn is correct. Remember that the conclusions you draw potentially could mean that you are going to propose that there needs to be a change in practice. So you want to have high confidence in the quality of your conclusion. 0:29:13.7
Slide 13: Example
All right, now I want to go back to that example that I've been using throughout all of the processes that we've been talking about here; and this is back to the temperature measurement, and remember the burning question: Do we have to do a rectal temperature on every baby that comes in to the emergency room?
Slide 14
And here is the evidence table, or just part of the evidence table, that the students put together based on their PICO definitions. You can get a little idea of the categories that they used up there by placing their data in the evidence table.
Slide 15
And here is the conclusions that they drew. They were able to find some meta-analyses that had been done. Quite a number of studies, actually, and it turns out that the rectal temperature is by far the most accurate, and it needs to remain being done in the emergency room. That is the conclusion that they drew.
Slide 16: Questions? Comments?
Page 1 of 1