Research Assignment

Slide 1: Analysis of Evidence Using a Synthesis Table

Hi everyone! This is the PowerPoint slide that goes along with week 11 content and this is on the analysis of your evidence using a synthesis table. So it is going to be a little bit … well, it is building on what you did last week, so bear with me a second, I will pull my PowerPoint slides up. Obviously, this is about the synthesis table but I am going to start with a little bit of information about the evidence table which we talked about last week because I want you to see the relationship between the two and how they build on each other.

Slide 2: Threats to Internal Validity

We talked about the threats to internal validity. The reason that we had a special lecture for that and the reason I did some testing is because understanding of these threats is a very important part of your looking at the literature that you are reading and making sure that it is indeed a good study. And the only study that you can look at and say that all the threats to internal validity have been managed is if you are reading a controlled randomized trial that has double blinding. And when I say “double blinding” what I am meaning is the interventioner does not know what intervention is being used and the subject does not know what intervention is being used. And so everybody is working without any information. If you have any other kind of design that you are using, there will be threats to internal validity and you will have to look at the way the researcher has handle that threat and decide whether you think it is appropriate or not. And remember the reason that we are worried about these threats to validity is because they could be an alternative explanation for why the findings are as they are. In other words, the intervention did not make the change but it was actually of one these threats to internal validity that we are worried about. So, that’s why I’ve asked you to really look at these and be careful when you are reading your studies. The other thing that I wanted to point out is that Level 1, according to Melnyk, can also include systematic reviews or meta-analyses. Now those can be poor; and whether they are good analyses or poor analyses depends on how the researcher has gathered the research studies. Sometimes you will find in those analyses when you read them that there are all sorts of different designs, not only randomized control trials but there will be qualitative studies and one shot case studies, all the things that you’ve learned, really do not protect against the threats to internal validity. It is very important when you read a systematic review, even though it is Level 1 and Melnyk says that it means it’s a very good study or it’s a very good source of evidence, it could be that the quality of the articles that were chosen is not up to speed, and then it would not be all that influential in you trying to determine which of your interventions is going to work out the best.

Slide 3: Evidence Table

All right. Let’s move on. Now, the evidence table is really the first step in the research synthesis process. And you use the evidence table really more than anything else to organize what you are doing. And I recommend that your organize by putting all the Level 1 evidence first in your table and then work your way down to lower levels of evidence and end with 7. When you do that, you then can have a kind of a gestalt about where is the quality of the evidence that you are looking at. I you have one Level 1 study and seven Level 7 studies, you’ll have to know that it is not looking good for being able to conclude that a new intervention needs to be used. Because remember 7 is personal opinion or expert opinion, it is not based on any research findings but rather just an experience of a group of so called experts. On the other hand, if you have eight or nine Level 1 studies, that’s very good evidence and you can look at those findings with a great deal of confidence. But now the thing that you have to look at is, now okay, “I’ve got all these studies that I can have confidence in the findings but are they studying exactly what the EBP questions is that I am looking at?” This will be a common issue that occurs when you search for your evidence and that is, there will be studies that are similar to or are very close to what you are interested in, in other words, what your EBP question is and what your PICO definitions are but it won’t be quite right, it won’t be just an exact perfect match. And when you run across one of those kinds of things, once again you will have to make a judgment on how close the research is to what you are doing and whether or not the finding actually add to the body of evidence that’s going to help you make a decision about whether or not … which one of these interventions is better or whether or not we need to put a practice innovation into place. All right, so let me just review what we talked about because it’s all about the evidence table and we’ll be moving on to synthesis table in just a minute. So the evidence table is your first step. It’s your basic way of organizing the articles that you have selected. And you are going to put them into your evidence table starting with Level 1 and then working your way down. And I talked last week about what would be typical kinds of headings in your table. In other words, those headings give you guidance on what you need to extract from your articles.

Slide 4: Evidence Table

And so the conclusions that you are hoping to draw from the evidence table are an overall gestalt about the quality of the evidence. And that will give you an idea about of how you need then to move forward. And let me just repeat then one more time, if you’ve got a large group of … if you’ve got a large number of Level 1 and Level 2 studies, that is excellent because what that means is that you’ve got quite a bit high quality evidence. And so what you need to do is look at the findings and find out is there a predominance of evidence and or is there great controversy. In other words, you’ve got about half and half of the articles - some found one intervention better than the other and other studies found just the opposite. That once again is a common situation that you may come across. Under those conditions then, there is not enough evidence to say that one intervention is better than the other or that a practice innovation needs to be put into place. After then you have done all of your work on your evidence table, it is time now to move to the synthesis … oh, one more thing about the evidence table. Yeah, I told about this. One other thing that I did not talk about the evidence table: each of you will create a different evidence table. You will have some similarities in your table but you will also have some differences. And the reason you have differences is because your EBP question is different. And so what you need to extract from a research study will be different for each one of you. So, realize that you will be modifying the potential model that I gave you.

Slide 5: Rapid Appraisal Questions for Randomized Control Design

All right. Now I want to go to the synthesis table. The synthesis table is really looking at different elements in your study. You’ve got the basic information but now what you want to do is capture the information that would allow you to say, “Oh, I’ve got great confidence in this study, I am really sure that those findings are valid.” And therefore, they get a lot of weight because it’s a good study, high level, and it’s got the important information in it. Your book lays out rapid appraisal questions for every one of the designs that you might run across. Well, maybe you’ll find a design that they don’t cover but they cover most common kinds of research designs that you are going to run across. And for each one of them, there is a different set of questions, and those questions then are what you use to make the row across the top of your table. And basically, what you are just doing – and most of these questions are “yes or no,” so you might do something like put a check for yes or leave it blank for no - and realizing that all of the questions that are being asked in this rapid appraisal form are critical to having a high quality study. And you look at these questions that I have on my PowerPoint slide right now … I have a fast finger here with the slides, sorry … let me back this up again. Look at these questions, a couple of them I want to point out. And what you are looking at is the random assignment of the intervention so that everybody in both groups, or how many groups you have, has the same opportunity to get the intervention. It is very important because that randomization will help protect against most of the threats to internal validity; because one makes the assumption that when you randomize that you’ll have equal characteristics of patients or subjects, pardon me, equal representation of subject characteristics in both groups. That randomization is very, very important. Then you are also looking at whether or not the healthcare professionals involved in the research are blinded to the study or not. Remember, blinding simply means that you don’t know what is being tested, you don’t know what’s being measured; and so all of that is done to try to get rid of human bias. And then how complete is the follow-up – you are looking at how long those subjects are being followed and is that length of time that they are being followed adequate. For example, if you were studying quality of life after becoming paraplegic, let’s call it, if you measured it at three months and called that an accurate measure of the quality of life, it will not be; because coming to grips with being a paraplegic and working your way through quality of life will take a lot longer than three months. So as we are talking about follow-up, what you are saying is that we are looking for how long the outcomes are being measured.

Slide 6: Rapid Appraisal Questions for Randomized Control Design

And then here are some more questions but also from the rapid appraisal questions for randomized control trial. Is the design appropriate? In other words, the way you figure that out is that you look at the hypothesis or a research question. And that hypothesis or research question tells you immediately what kind of design you should use. The other thing that’s so important in any kind of randomized control trial is the a priori calculation of the sample. In other words, you figure out how many subjects you are going to need based upon a power analysis. So you are looking for a power analysis and once they determine how many subjects they are going to need, did they actually get that many subjects. That makes a big difference. Then, what are the inclusion and exclusion criteria? We talked a little bit about that and you actually have to write that up in your EBP wiki that’s coming up because what a researcher chooses to include and not include can really influence the findings. In some studies that I was doing with pregnant women and I made the cut off point between 18 and 35 because at 35 a pregnancy becomes very different than it is when you are younger. And so they really are not homogeneous groups when do it that way. That is the kind of thing that you are looking for in the inclusion and exclusion criteria. Most frequently, the problem is they’ve included groups that really shouldn’t be there because that group is going to alter the findings, so that you can’t be confident in them anymore.

And, then finally, could the choice of the participants influence the finding? This is selection bias, remember, that we talked about. This is intervention selection bias. And you can choose groups of participants that have characteristics that will assure that you have the outcome you want. And you have to really be careful about that. And I know I have said that in my announcements and I’ve told it to you personally in emails, just because it’s written doesn’t mean that it’s good. So you always want to keep yourself alert to things that aren’t done well. And you can really just use your common sense. And the whole deal when we are looking at in any of the research studies: you want to know, can I have confidence in the findings? And the way you know if you can have confidence in the findings is, are there any of other things going on in this study that could compromise the findings? And that’s what all of these questions in your rapid appraisal are asking you to get at. And then when you move on down the levels of evidence … You know, Level 1 is randomized control trials that have been put together into synthesis or meta-analysis. And then Level 2 is individual randomized control study. That Level 1 and Level 2 are … should be excellent articles but you really have to look at them and make that determination on your own. Then you start getting into the less well controlled studies and while you may find the study very nicely done with very few threats in it, it is going to have by definition less weight than the randomized control trials. For example, a cohort study is going to have less weight because it doesn’t have all the protections that a randomized control trial does. But we use all of those different kinds of designs because in health care research when you are dealing with human beings, there are many times when we actually can’t do a randomized control trial. And the way you can get out the information that we want is through other designs. It is not that they are bad designs, they are essential designs; but the whole deal is that - can we have confidence in the findings? And the other thing that you want to make sure that you have on your rapid appraisal form that you are using for each study, or actually this is going to be on your evidence model … your evidence table, is exactly what those findings are. Of course, eventually you will have to make a decision about things. Oh, that’s another slide, let’s see …

Slide 7: Rapid Appraisal Questions for Randomized Control Design

All right, here are some more questions that you need to ask. Oh, this is a very important one: are the outcomes clinically relevant? Well, we can have a really great research study that has been done, great controls in it so that you don’t have any threat to internal and external validity and very significant findings and it turns out it’s statistically significant but not clinically significant. And I had exactly that kind of thing in my dissertation research that I did, where I was using a relaxation technique to decrease blood pressure. And I did get blood pressure decrease and it was statistically significant but it was 2 millimeters, that’s not clinically relevant. So it’s those kinds of things that you are really going to have to look at and make sure that it makes any clinical difference; because if it doesn’t make clinical difference, there is no reason to deal with it.

Slide 8: Rapid Appraisal Questions for Randomized Control Design

And then finally, after you’ve looked at all the quality of it, you have to ask, “Would I change practice based upon this finding? Of course, you always know that one study does not a practice change make. But if you are getting a number of the studies that are having the same findings, most of which are high quality evidence, then you begin to think that you can actually make a decision to put some other kind of intervention into place. And here are some more questions that have to do within your research support.

Slide 9: Synthesis Table

All right, now … I think I kind of talked about this before but let’s go over it again. Now, your synthesis table then is a way of organizing the more specific quality level questions that you’ve asked about each one of the studies. And so, on the row at the very top, you will have the questions - now, you may just want to number them questions da-da-da or you may want to abbreviate what they are so it makes it a little bit easier to put them in the table - so that when you can see yes’s or no’s or checks or no checks in that table, you’ll have another sense of the quality of the articles that you have. The whole reason that we do a synthesis table or an evidence table is to be able to determine quality because quality determines our confidence in the findings; and confidence in the findings determines whether or not we are ready to try this new intervention in practice. So you want to be looking for that. And on the first column to the left, is where you put all your studies. And once again, I would highly recommend that you put those studies in the same order as you have them in your evidence table. That way when you put the two tables in front of you and look at them, it really gives you a very good idea about how much confidence you can have in the findings. So once again … I guess, when you do your first evidence table before you fill anything in, just copy it and you’ve got it ready for your synthesis table. Then have all your bibliographies from your articles down the left hand column. And then in the row across the top, you’ll have the questions that were used in the rapid assessment audit. So you can see that you need to bunch things together because you are going to have different questions for different designs. So, the systematic reviews and … the systematic reviews will usually contain control trials. Then in Level 2 you‘ve got single control trials. Both of those will have the same questions across the top of the synthesis table. And then when you move on to Level 3, where you are talking about some cohort studies and … I think cohort is 4, actually; you are doing quasi-experimental in Level 3. So what you’ll need to do is put like things together and you will have a number of synthesis tables because the questions are different for each design. So the rapid assessment … rapid appraisal questions are different for each level or each design. But you still will be able, I think, to lay them across from the evidence table and begin to see … make a determination about how much confidence you have in the findings. And that is the information that you use in your wiki to argue whether this intervention should be implemented in practice or whether there should be a practice change, or whether one intervention is better than the other. And so that will be a very important part of your wiki to use the data that you have in evidence table and the data you have in the synthesis table to draw that final conclusion about whether or not there should be a change in practice.

Slide 10: Questions?

All right, now we have come to the end of my presentation and I am thinking that you very well may have a number of questions about what I have just discussed and so let’s have a good discussion going on the discussion board under the forum Questions about the Course so that you will be able to talk to me about your understanding about what I had in lecture and we can dialogue and so you really begin to internalize it; so when you look at these studies you know that you know what you are doing. I will keep an eye on the discussion board. I’ve been going there very routinely to see if anybody has been leaving me notes. I haven’t had any for a long, long time. So I hope that you will participate and we can talk about final conclusions because when you look at the grading criteria for the EBP wiki, you will see that that is what I rate the highest. I want you to argue for why you are making one conclusion or another. Use the data that you have gathered to convince me that you are making the right decision about changing practice or not changing practice. So I hope this has been helpful, and we are moving on toward the final presentation of the EBP and where you can put together all of the skills that you’ve been honing throughout this course. So I am looking forward to it and my job is to be here to help you. So please talk with me, ask me questions, ask for more explanations if need them so I know what it is that you need. If you don’t talk to me, I don’t know what it is that you need.

Page 1 of 1