Psychology week 6 assignment 2

program evaluation. This week we're going to work on data analysis, and then next week we will be bringing everything together and also working on adding the discussion section. So you'll be bringing everything together in your week seven report, and then there's just a discussion in week eight. So we have the light at the end of the tunnel, but now is not the time to slow down. I know it can be hard after a holiday weekend, but we still have the data analysis and pulling the whole report together. And then we'll have that final discussion in week eight, and then And you'll have a nice three -week break. So that will be well-deserved time off. So hopefully you guys are feeling good about all the progress you've been making. And I'm sure it's, you know, everybody's ready for the break by the end of the year. Okay, so last week you guys created a mini data set with just 10 participants. and that was really just an exercise to help you think about your data and you know your variables that you're going to be looking at but we're not going to be using that for our assignments so there is a data set in the week six assignment area and I'm going to go over the assignment but I just wanted to make clear that I'm giving you a data set so that you know we don't have to worry about problems with a data set that that you guys created like overall i think people did a good job but we would need more data than 10 participants and some of the data sets didn't have pre-test and post-test data and we're trying to look at changes right we're trying to see if a program evaluation is effective so we need to have both pre-test data which is at the beginning before the program starts and then at discharge. So that's our post-test data. Okay, so getting started for this week, we're going to have a discussion. And your discussion helps to prepare you for your assignment this week. So we've had that before. We've had a discussion that helps you prepare for the assignment. Um, but this one is, is definitely one that, you know, should help you with your assignment to think through some of the things you're doing and why. So do your discussion earlier in the week, your initial post is due Wednesday night. And then, um, you know, hopefully that'll help you better understand what you need to do for the assignment. I'm going to talk through like the logistics and I will demonstrate how to do some of these things. But I'm not going to tell you the answers to these things about what you do and why. So some of this is just to help you think about what kind of statistical analyses you'll be doing and what the purpose is of your analysis. So the first part of the discussion says, explain the statistical analyses you'll use to analyze the data in order to answer your evaluation question. So my first tip for you would be to start off by saying what your evaluation question is. So you can't state what kind of analysis you're going to do if you don't know what your research question is, right? That's kind of a basic research methods 101, right? We need to know what are we studying? What are we trying to examine? So I would start off by saying what your research question is that you're trying to evaluate. Next, in order to figure out what kind of analysis you're going to do, you need to think about what kind of data you have. As I mentioned before, you're going to have both pre- and post-test scores, but you should look at the data or the codebook and try to figure out, are these pre-test, post-test scores on a continuous scale, or is it categorical? And that's going to help to guide what kind of test is appropriate, you know, is what kind of measurement you have. Okay, so then that's going to bring us to the second bullet point here. So next, I want you to think to answer this question, how will you determine which statistical analyses you'll use to answer your evaluation questions? so because we are trying to look at change like if there's any change between pre-tests and post -tests depending on your research question you know it's whether you're looking for an increase or whether you're looking for a decrease depending on what kind of measure you're you know you're you have you would need to explain how you would decide between the two main kinds of tests that look at the difference between a pre-test and a post -test. So would you be using a paired sample t-test or a Lecoxon signed rank test? And how would you decide which of those you would use? So, you know, as you will need to do in your assignment, you are going to need to check for assumptions before you can decide whether you're going to use the paired sample t-test or the Wilcoxian side rink. Can you guys still hear me? I see that my video just froze. I can. Okay. So it's just my video. I'm going to turn my video off for a minute then and see if that helps. That's kind of weird. I don't know what's going on with my computer. Okay. It's not even allowing me to turn my camera off, which is kind of bizarre. um okay well i'll just keep going then um so after you you know discuss what you're gonna you know assumptions you're gonna check like normality you know and things like that you know that will help you decide whether you do a paired sample t-test or wilcoxia test you should discuss any data cleaning steps that you need to do so for example looking at missing data outliers or unusual values just to make sure your data set is in order and you understand your data. And, and then you should, you know, explain why, you know, or what kind of reasoning you would use to decide whether to use the paired sample t-test or the Wilcoxian sign test, not just which test you're going to pick, but like how you'll decide. So you don't have to add the answer yet about which test you're going to use. I just want to know that you understand why you would choose one test over the other. And then for this fourth bullet here, it says, what would be the best way to present the results in your report? So obviously I want you to report statistics, but you also may want to consider using some tables or graphs and so you know think about what kind of tables or graphs would be useful to make a pre post chain post test change easy to interpret like what kind of graph simple graph you might use or a table you might use and then finally your analysis section should just help set up how your results will speak to program effectiveness. Like how is this going to help you determine whether the program was effective or not? Okay, so this is really just kind of thinking through the analysis that you're going to do for your assignment this week and hopefully helping you to think through a little bit more about why you might do certain analyses and one over the other um all right so before we get into the the details about the um analysis which is your assignment um do you guys have any questions about the discussion so far hi cameron Hi, Katrina. I see you guys came. No questions here. Sorry. That's okay. Okay, so we'll keep moving along. So there's two parts to the assignment this week. So one part is going to be the second part of your method or like another part of your method section, which is the analysis section in the method section. And that's where you're going to just provide a description of what stats you're going to use to analyze your evaluation question and what variables you're including and what kind of data cleaning or screening steps you're going to use, like check for missing values, check for outliers, assessing normality, and your rationale for choosing, you know, how you're going to choose whether to use either the paired sample t -test or the Wilcoxian sign rank test. So the discussion clearly leads into this part of your assignment. And you can also refer to page 222 in your textbook. That, you know, can give you some examples. The second part of the assignment for this week is to actually do your analysis and write up a short result section. So you're going to use the data file that's in the assignment area. And I just want to reiterate, I said this at the beginning of class, but I want to reiterate, we're not using the data file you created in week five. That was just hypothetical data. And I want everybody to use the same data set. So we're going to use a data set that's within the classroom. I created a data file that has all of your, or a bunch of possible variables. I think that, you know, if there's, I think I've talked to you if your variables are not in the data set, but if they're not, let me know. there were i think there were two people who either posted late or um or maybe there was a little bit of confusion so um but when i created the data file the the all the possible um pre and post test scores were in there so i'm going to bring that up so that i can kind of go over that with you guys so you guys know what I'm talking about. So let me see if I can find the code book that I put in the classroom. So in this data file, there are a number of variables. There's an ID number, age, gender, race, ethnicity, number of treatment weeks. So, you know, if you were doing some additional analyses besides the simple one that we're doing for this class, you might use some of these other variables. But for your purposes, what I want you to do is just find your pre-test post -test pair. So for example, some people will be looking at the SDQ pre and post -test, which is the strength and difficulties questionnaire that has values ranging from zero to 40. Some people will be looking at the CANS domain total score has scores ranging from zero to 30. Some people are looking at the youth outcome questionnaire with scores ranging from 0 to 200. Some others are looking at the PHQA depression scores, pre- or post-test scores ranging from 0 to 27. Still others are looking at the anxiety measure, which was the GAD7, which has pre and post test scores ranging from zero to 21. And then finally, there was the perceived stress scale for children, and that has pre and post test scores in it as well. So there are, I guess, six different outcomes, and you would just choose the one that most, that best aligns with your research question and use that to analyze your data. Okay. Does anybody have any questions about that? Because I just want to make sure that's clear that you should use, you know, one of those pairs of data. Yes. Yep. So you're going to check your assumptions and everything with that data set. So I'm going to go through the steps of the assignment. I'm going to, let me just run that, run through that right now. But, yes, with that same data set, I want you to go through these steps. So let's see. It's a little small. Let me see if I can enlarge this. So you'll use that week six data set. It's called Program Eval. And you'll be looking at, you know, descriptive statistics. you'll be doing you know you'll check for missing data you'll check for outliers you'll check for normality so those are the things you're going to check and then you're going to run either the paired sample t-test or the Wilcox test and then you will create a graph and you know you'll summarize those results. So I'm going to go through an example of this, but you should be doing everything with that data set. I don't want you guys to use the week five data set. Any questions about that, about what data set you're using or what variables you're looking at? no okay did i add a question so this is the methods section so we don't want to do too much background about the data variable we don't want to like talk too much about the the measure that would that this is mostly just the results right so there's two you're writing you're probably writing like two paragraphs one's going to be in your method section and it's going to describe what analyses you're going to do and what your assumption checks are going to be like what you're going to check and do and then the second paragraph will be you know, under a subheading called results, then you'll just write up the results, the statistical results, and, you know, include a graph. But no, you shouldn't be talking about the measures there. That should be in a different part of your report. So let me see. Thank you. Yep. So I think it's. Yeah. So in I think it's in this section, outcomes and measures. And so right under that is this new section called data analysis. Now it's like too big. I can't see it. But yes. So in the outcomes and measures, that's where you talked about your measures. So here you're just going to have your new paragraph about data analysis, and then you'll have your results section. So you're just adding to this report. And then next week, you're going to revise everything based on all the feedback you've gotten, pull it all together, and then add a discussion section, just like you would in, you know, any other research report. you know, I don't know how much experience you guys have had with that, but when you do your dissertation, obviously it's going to be a much bigger, much more fleshed out thing, but you'd have a, you know, chapter one's kind of your intro and background, chapter two's your lit review, chapter three's your method, chapter four's your result, and chapter five's your discussion. So we're kind of doing all the different pieces of the report. Okay. So, if you guys are ready, we can go ahead and take a look at the data and we can can demonstrate we can look at oops at the different you know at the different steps that you should be taking here so any questions before we start going into the into this so i'm gonna i'm i've started putting together a guide for you guys but i'm gonna wait to put that in the classroom but um until after you guys have done your initial discussion post. So I'm going to walk through this, but I will also give you a guide later in the week, just because I know sometimes it can be challenging to do some of these things the first time. Okay, so the first thing we're going to do is we're going to look at whether there's any missing data and you know that's something that you would just be reporting so let me see if I can share that screen with you okay so this is the data file I opened it up as you can see you know the data is all in here, SDQ pre and post, CANS pre and post, YEQ, PHQA, GAD, and perceived stress. I also created some different scores because, you know, with a pretest post-test evaluation, you're really looking at what's the difference between pretest and post-test. So this variable here just subtracted the post-test score from the pre-test score. So it's just how many points difference was there between pre-test and post-test. So the first thing we're going to do is look at if there's any missing data. So I'm going to go to exploration descriptives. So what I want you to do is look at your pre-test and post-test data and report how many participants there were and and how much data is missing so for example if I was looking at the perceived stress pre-test and post-test scores I could see that there were a hundred people who completed the perceived stress scale pre -test and 97 people who reported the post-test. So there were just three people missing. So we're not going to do anything about that. We're just going to let it do list-wise deletion. It'll just, you know, delete those people from the analysis. Um, but we could just report, you know, there were three, three participants missing from the, you know, post-test data. Um, next we would want to, I'm just going to, um, refer to my guide here. Next, we were going to look for outliers and see if there are outliers. And so what you would do then is go to the plots and check box plot and you could have it label outliers. So in SPSS or Jamovi, it'll label potential outliers. Like here it has, you know, case 40 and case 27 are potential outliers for the pre-test and 40 and 27 are potential outliers for the post -test as well. So there are two potential outliers. If you do find potential outliers, what I would like you to do is also look at skewness and kurtosis. So I'm going to check those boxes off here, skewness and kurtosis. And if your value is here, we can see skewness 0.426.391, kurtosis 0.239 minus 0.0143 i don't expect you guys to necessarily remember this from statistics class but the closer to zero skewness and kurtosis are the better so because these values are less than three that means they're not they're really not extreme they're in fact they're all less than one or less than negative ones so those you know are all fine so we're just going to leave potential outliers in there, especially because these are not too extreme. So that is our second step. So we just went through kind of quickly missing data and outliers. Before I go on to the next step. Does anybody have questions about those first two steps? So I want you to just report missing data and report potential outliers, but not do anything about them. So yes, you would just be running the analyses with the missing data. What SPSS or JMOVI does is if you have two scores you're comparing, like with the paired sample t-tests that you're looking at, like pre-tests and post-tests, it'll just eliminate those people who don't have both a pre -test and a post-test score. So that's called list -wise deletion or pair -wise deletion. And it'll just, you know, it'll, you know, get rid of rid of those. Any other questions about looking at missing data or outliers before we move on to the next step? OK, I'm going to keep moving because I don't never know how long Jamovi is going to work for me because I use the free version. OK, so the next thing we're going to do is we're not sure if we're going to use a paired sample t -test. but I'm going to use it to check my next assumption, which is normality. I'm going to use a different one just so that I'm not only looking at perceived stress. So I'm going to look at anxiety, the GAD7 pre-test and the GAD7 post-test. So that's paired sample. Now let's just ignore this output right now because we're not, I don't know if we're going to be able to use this test. What I'm going to use, what we need, our next assumptions we need to check are these, this little part here at the bottom, it says assumption checks, and I'm going to check off normality test and QQ plot. So that's going to help me determine if I can use the paired sample t-test. So the difference between the pre-test and the post -test, it's saying the normality test, the Shapiro -Wilk is a value of 0.914 and p less than 0.001. So if it's p less than 0.001, does that mean it's significant or not significant? It's very significant. It's significant. Yep. Yes. So we would say that's significant. Do we want a normality test to be significant? No. No. No. So that means this is not normally distributed. Now, the other indicator that it's not normally distributed is our QQ plot. If it was normally distributed, these dots would more or less line up against this line. Now I put a document in the classroom under the week six overview that gives you some more information about the QQ plot in case you don't remember or you're not that familiar with that. But yeah, we would want this to be more the dots lining up belong here, not, not, oops, oops, I just lost a movie there. Oh, gosh, okay. Anyway, we did see that that data set was not normally distributed. And so if you have, if your variable, whatever it is, if that normorality test of Shapiro-Wilk is less than P, less than 0.05, and your QQ plot does not, you know, not looking like it's fairly much, you know, lining up with that line, that diagonal line, that that's an indicator that it's not normally distributed, okay? So that means you would have to do a different kind of test other than the paired sample t-test. We only use the paired sample t-test if our data is normally distributed. So if your p-value is greater than 0.05, you can use the paired sample t-test. But if it's not greater than 0.05, then we're not going to use the, we're going to use the Shapiro-Wilk instead. Okay. Questions about that? Are you guys still able to see my screen now? I'm just trying to get this data file back. yes okay great okay so we um so what do we do well if we if our data is normally distributed we're going to use the paired sample t-test i'm going to show you some things that you should check off here too if if you're going to use the paired sample t-test so So we would want to check off, like, some additional statistics here, like the mean difference, the confidence interval, and the effect size and descriptives. We want to be able to report all these things if we're using the paired sample t-test. So that would give you, checking off those things would give you all the statistics you need to report other than your plot. We want to also have a, you know, some kind of graph to illustrate this. So I just checked off descriptives plot down here, and that gave me this nice little graph that's showing, well i don't know if this is significant or not i'm not going to say because i don't want to give the answers away but this is looking like there's quite a big difference between the pre -test and the post-test in terms of scores do we want the suq scores to go down i don't know that's up for you guys to decide if if that was something you were looking for that to decrease or increase increase or decrease but you know if it's not going in the direction that you're predicting then this would not be a great thing but it does look like there's quite a difference here so that having that graph would be useful as well because that would give you the information you need and then obviously with the t-test you want to report your t-statistic with the degrees of freedom the p -value um the effect size which is cones d and um you know the if if you're using the paired sample t-test you would report the means you know the mean for your pre -test and post-test along with standard deviation so hopefully you still have notes from when you took stats or you can look something up online for reporting t -tests, but this is all the information you basically need. Now, what do we do if it's not normally distributed? Well, if it was not normally distributed, like I said, we would use the Wilcoxian rank test. So that's just a check. You can do it two different ways. I'm going to show you the two different ways you could do it. One way you could do it is just within the paired sample t-test, you just click Wilcoxian rank, and it'll give you that output down here. Um, and, uh, I'm going to, uh, so right here in the same paired sample t-test table, I can see the Wilcox W statistic. It says four, five, four, nine, um, and it reports the mean difference. and so you know a bigger the bigger the w value the you know stronger the result so again if your data is normally distributed use the paired sample t test if not use the wilcox rank test the other way you now when you use the wilcox rank test we would report the median and standard error for both the pre-test and the post-test. So when your data is normally distributed, we use the mean and standard deviation. When the data is not normally distributed, we use the median and standard error. So that's the basic rundown of that. Now, there is one other way you could do the Wilcoxian test, which, you know, it's up to you, which is easier for you to interpret, but the other way would be to do a one-sample t-test. And that would be where you would look at the difference score. And so you could look at, you know, something like the, like, let's say we wanted to look at the PHQ, different score, we could um then you could do that you know you could do the test um that way as well but it's probably easiest just to kind of be consistent and just use the paired um sample t-test and click off the wilcox test um so that we're doing it all the same way um all right so I went through a lot of information. I just kind of wanted to give you guys an overview. As I mentioned, I will give you some written directions as well later in the week. But right now, I want you guys to think through what you're checking, like what your assumptions are for these kind of tests and, you know, what it is you're trying to explore. What are you trying to examine what are we looking at with this program evaluation like what are you trying to figure out what's your outcome what are you trying to determine how do you determine if it's effective and what do you you know what are you looking at so that's what you're going to do with the discussion and then for the analysis you'll you'll do that for the second part for your results section. No problem, Katrina. Thank you for coming. All right. So any questions? I'm happy to also answer any individual questions now, too, if you have any questions about your report or your outcome, or just, you know, about the discussion or analysis in general. Okay. All All right. Well, I think it's hopefully pretty straightforward. I tried to kind of make it straightforward that you're just focusing on one outcome. And so you can really just look at, is there a change from pre-test to post-test? And so hopefully that'll be pretty straightforward. And then, like I said, program evaluation is kind of cool because it's something that, you know you kind of like are focused on the same thing from start to finish and it is something actionable where you know there are jobs in the field doing this type of thing if it's something of interest to you um all right well i'm here if you guys have any questions if not i hope you have a great week and i will see you in the classroom thank you have a good night thank you dr kelly you too thank you thank you good night