week five discussion program

my powerpoint up here okay welcome to week five of program evaluation rsm 816. so we are now officially past the halfway point in the course so hopefully you guys aren't feeling good about your progress so far you've created your logic model your introduction to your program evaluation and two parts of your method today we're going going to talk about the final part of your method section, which is your analysis plan. Well, actually, no, that's next week. We're going to do that next week. But we are going to start on your creating a data set. And so next week, we will focus on the final part of the method section, which is your analysis plan. And you will also carry out your analysis and write up your results. So then you will have all the parts of your report other than your conclusions. And in week seven, you will pull all those parts together and write the conclusions. So you'll have your full report so you can incorporate all your revisions and anything else and have your full report together. So this week, we have an important part, which is working on your data set. So this is a little bit different in that it's a fictional data set, but I just wanted to give you guys experience with this because, as I said at the beginning of the course, program evaluation is an actual area where you could potentially get jobs with your degree so I kind of wanted to give you some experience with the different parts of creating a program analysis and obviously writing up a program evaluation report and working with data as well so So before I start going into our discussion for this week, are there any questions at this time about program evaluation or what we've been doing so far with our reports? I just got a little bit switched around. So we are making the data set this week or? yes and that's the data in yes we're going to put data in so um last week you guys chose your measure and so now that you know what your measure is going to be we could we could work on you know entering our data so we're not collecting real data we um you know we're going to create a fictional data set. Um, but, you know, basically there were, you know, from my reading of, um, the discussions from last week, it seems like there are, um, six measures that were chosen. Um, And I'm going to see if I can actually put them in here because I think that would be useful to add in. I will put this in the discussion, too. But it seems like there were six. If I can pull my box out. There were six measures chosen. Can you see those, Cameron, on the screen? Yes. Okay. So it seems like there were six measures chosen. I'd like to keep it to these six because I am going to give you actual data in week six. So this week you're creating a fictional data set just to get experience with that for your discussion. But I will give you actual data to analyze next week. And so I'll give you a data set that has these data for these six measures. Okay, so that you have actually correct measures. Obviously, hopefully you guys will do a good job with yours, but yours will only have 10 items. And I think it's just going to be more streamlined if I actually give you a data set next week. So this week, you will be creating a fictional data set, and you will use the measure that you chose, whichever of these six measures you chose for your evaluation to measure your outcome. Is yours in this list, Cameron? yes and i'm so geez i i was looking forever and and finally last night i found the full text of the measure with all all 130 136 questions oh that's a lot of questions yeah that's one of them all like it goes on for two or three pages so oh wow very cool that's great well that'll come in handy because I'm going to go over what you need to know about your instrument um um so again for your discussion I've changed this this week to be only a discussion because it is Thanksgiving week and I do think it it create it is it does require a fair amount of effort to create a data set. So that's kind of a bigger discussion anyway. I believe it is worth a little bit more points in terms of scoring than your typical discussion. But regardless, it is a useful assignment because it allows you to really think about your data. And so what you're going to be doing this week is creating a data set that has 10 hypothetical cases, like 10 hypothetical participants based on the outcome measure that you selected in week four, okay? So, but in order to create a reasonable data set or something that seems to be real or that could be possible, you need to know your instrument. So Cameron, I'm glad you found your instrument. Okay, so let's go through this, what this entails. So this week, you're going to, you know, pretend collect data. But you're, you're not really collecting data, you're just going to create a database based on the measure you selected and include various information about that just to show that you really understand your measure. Okay, so the first part of your discussion involves a brief description of your data set. So first you would talk about what your selected measure is. I'm going to use as an example here the perceived stress score for no other reason just that I've had a lot of familiarity with this. I've used this in another class of mine to demonstrate reverse scoring, and I also have had several students use this for their dissertation because it is a well -established measure for stress, and it's only 10 items, so it's pretty simple and straightforward, unlike your scammer that has a lot of items, but I'm going to just use it as an example. So here, part one, I'd like you to give a brief description of your measure. So if I were saying, you know, I'm going to use the perceived stress score, and I'm going to talk about scoring rules and the variables that we created. So for all of you, what I would like to stress is that you only need to do the summary score in your data set. So Cameron, you don't have to create 136 columns, you know, with each individual item. Obviously, typically, you know, we might want to have all the individual items, particularly if we were creating subscales, or if we wanted to, you know, analyze something about the different, you know, individual items. But for the purposes of this assignment, you're going to just be creating a summary score. So I don't know if like a summary pre-test score and a summary post-test score. So I don't know how familiar you are with that yet, but you would need to report whether there is any reverse scoring. And what do I mean by that? Well, in the past, let's say 20 years ago, there used to be a common belief that there should be some items in the affirmative and some items in the negative. but as we learned in psychometrics this line of thinking has changed and now it's considered to have all your items in the same direction like all affirmative or all negative but there are many instruments that are still very good instruments that do have some items in the positive and some in the negative and what happens is if you have some items in the positive and some and the negative, we need to do what's called reverse scoring before we create a composite measure because we want to have them all going in the same direction so that we can add them together and sum them to come up with a score. So I'm going to pull up the perceived stress scale that I told you I was going to use for the demo today because that does have some reverse scored items. So the perceived stress score is only 10 items. They're scored from 0 to 4. And there are some items that are in different directions. So for example, question one says, in the last month, how often have you been upset because of something that happened unexpectedly? So if there's a positive score here, that would mean like higher stress. Now, what's reverse scored or in the opposite direction is, as an example, is question four. In the last month, how often have you felt confident about your ability to handle your personal problems? Now, here, a high score would mean low stress, right? So we would need to reverse score this positively worded item because most of the items are negatively worded. So like what I mean by that is in there negatively worded. And if you have a higher score, it means more stress. So here at the bottom of the instrument, it says reverse score questions four, five, seven and eight. So if you have reverse scoring, this would be discussed in your instrument, you know, like here it's at the bottom of the questionnaire. So you would need to report as part of question one that items four, five, seven, and eight are reverse scored. Now, I don't think anybody in our class is using the perceived stress scale, but I believe one person is using the perceived stress scale for children. And that's a different questionnaire. So, you know, you would whoever is doing that would need to look at that questionnaire and see if that I know if that questionnaire has reverse scoring and, you know, report on that questionnaire, not this questionnaire, because this one's for adults. So here we would need to reverse score questions four, five, seven and eight. So a zero would become a four. A one would become a three. A two stays a two. A three becomes a one and a four becomes a zero. and now and then we would also need to know like what's the possible range of scores like what's the minimum possible score and what's the maximum possible score so let's go back and look we have 10 questions and the scores range from zero to four so it's possible that every that there's somebody who said never for everything so if we take 10 times zero what would our lowest score be our lowest score then is going to be a zero right so our minimum possible score is a zero now on the other hand it's possible somebody said um four to everything right so we could say very often to every question. So, well, they said very often to all the negatively worded questions and zero to all the positively worded questions. So the maximum possible score is 40. So if we have 10 questions times four, it's 40. So you would report in part one that your minimum possible score is zero and your maximum possible score is 40. Okay, so we need to know this before we create our data set, because you don't want to be creating scores that are like 65 or 80 or whatever, because you'd be like, well, that's an obvious outlier. Well, it's not even an outlier. It's just an inaccurate data point. It's not possible to have a score of 65 or a score of 80 when your range is zero to 40. Another thing you should be reporting in this part one is whether higher scores indicate better or worse functioning. So in this case, if somebody has a score of 40, do you think that indicates a better stress score or a worse stress score? These are negatively worded. So if they have a score of 40, that means they have a worse stress score, right? So that indicates worse functioning. Okay. So the reason why we do that is so we know what our possible scores could be so that we're not coming up with impossible numbers. So we need to think about the possible scoring range of our measures and whether there's any reverse scale items. Okay, so let's go back to this. So you could also report here under the second bullet point whether there are subscales but we're not going to create those it's going to be too much work particularly if you have an instrument like cameron does with 100 some items because that would there's probably several that tells me there's several subscales um because that would be a lot of different subscales do you do you remember offhand how many subscales your instrument it has, Cameron? Yeah, 13. Oh, wow. Yeah. See, I knew there were a lot with that many items. I was like, yeah, that's got to be a lot of subscales. Okay. So, again, three things we're going to report here. We're going to report, I guess it's maybe a little more than three, but you're going to be rewarding your measure. Whether there's scoring rules, reverse scoring, like for the perceived stress scale, I would say there's 10 items and there was reverse scoring for items 4, 5, 7, and 8. And then I'm going to report the range of possible scores. So if perceived stress, the scores range from 0 to 40. And then I would report if there are any subscales, but I don't have any. There's only 10 items, so there's none in this particular case. But again, you don't need to create the subscales or do the scoring for the subscales. I just want you to report if there are subscales. And if so, how many? Like Cameron's has 13. Okay. So then you're, another thing you're going to need to do is create your data. And we can go through there are videos in the classroom going over this but I want you to think about what item or what level of data you're creating so you're not creating item level data so we're not entering in 10 you know different items for the stress pre-test score and the 10 items for the stress post-test score or the 130 some items for Cameron's measure for the pre-test and the post-test we're just going to create for learning purposes we're just going to create pre-test and post -test total scores so you'll have an ID number for each participant their pre -test score and their post -test score and you're gonna have ten rows so I'm gonna go through all these parts and then we can go through um setting setting this up um i just um so make sure you under before you do this though make sure you understand what a total score looks like you know what that's your summary score so again for perceived stress your total score would range from zero to 40 um some of them have you know, more complex rules than others, you know, but you can, you know, you know, make up something that, like, works with your data set, but hopefully, do you know, Cameron, which measure are you using? Is it the SDQ? Yes. And with the SDQ, do you know if they have a total score or just subscales? They have both. Okay, great. So, yeah, so you would just be creating your total score for that. Okay, so you'd have like pre-SDQ and post-SDQ. I was thinking I was going to include like 20 questions and then just do two of the subscales. yeah i mean you could if you want but i was trying to simplify it and just say just use a total pre-test score for sdq and a total post-test score for sdq but if you prefer to do it for one of the one or two of the subscales that's fine but when i give you the data set next week it's just going to be a total oh okay i can just like discuss some of the questions as an example yeah and i mean when you get the data next week you could pretend it's a subscale if you wanted if there was one particular subscale you were interested in oh i don't know what's going on with my camera today i just shut it off because it's saying it just froze me and i got some kind of warning message okay there it came back all right um so so i'm just trying to keep it simple um yeah i hear you you know what i mean i don't want to over complicate it um so when you're creating hypothetical data i want you to try to think about you know, your pre-test and post-test scores. And I want you to think about what you might reasonably see. Obviously, we want to see improvement, whether that's an increase in values or a decrease in values. Like for stress, we'd want to see a decrease, right? Because a higher score means higher stress. But we want to have some reasonable data. So post -test scores often improve, but not always, and not every participant is going to improve. So you know, you can have some higher scores or lower scores and some missing values. um you don't want to make your data too perfect um so just think about whether your data would potentially you know be realistic um in terms of what you're seeing you're not going to see like perfect patterns for every every participant um you should obviously use clear naming for your variables like I'll probably use PSS pre and PSS post that's perceived stress scale PSS is the common abbreviation um and then obviously we will after we have our data set we're gonna then go on to the next part, which is that we would have some, what do you call it? Descriptive statistics. I don't know what is going on with my screen today. So it's like frozen again. in. Okay, here we go. So then we, yes, after that, we'd run descriptive statistics in JMOBI or SPSS, like mean, standard deviations, or frequency tables. And then finally, I'd like you to write up a short paragraph explaining what your descriptive statistics say about your hypothetical sample. Obviously, we're only having 10 participants, so it's not a whole lot. It's just kind of a little snapshot. And, you know, we're going to, it says anything you need to double check before week six, we're not, I'm going to actually give you a data set in week six. So, but that would just be like, well, what would I need to look at if, you know, or double check if I was really going to use this for my week six data set. So those are the basic steps or components that I want you to be thinking about. And, you know, like, is your data set like suspiciously small? Or is, you know, is it, you know, is your standard deviation really large, suggesting an outlier, like, you know, that type of thing. So just check that out. So before I go into the data set, I want to just remind you, I did post this in the classroom, but I just want to remind you guys that this week, the discussion is still due on Wednesday at midnight. the reason for this is that that's the time it's always due but more importantly it is a holiday weekend and technically um I have off Thursday through Sunday that's you know the university holiday but because I have some time off I wanted to give you guys some time off too in the sense that you don't have an assignment and if you can complete your discussion by Wednesday night then you get the long weekend off right so um but I also wanted to provide some flexibility because I know some people do let's say a lot of cooking on Wednesday and I don't have time to do this um before 11 59 on Wednesday so if you turn it in um as long as you turn it in by Sunday I'm not going to penalize you if you turn it in late, because I know it's a holiday week and people have different schedules. I know, you know, just from I've been teaching for a long time. I just know that some students say, oh, I'm just Wednesday. I'm just busy grocery shopping, cooking, whatever, getting ready for the holiday. So there is flexibility. I, however, will not be available over the weekend. so if you run into questions please you know get them to me by Wednesday at five o'clock if you run into issues or if you think you're not going to look be able to do this until the weekend and you think you'll have questions you might want to set something up with tutor .com all Kaiser students get five free hours of tutor.com through Blackboard that's just a link you click on I think it's on the home page but I think it's also in the classroom like it's just one of the tabs you can click on have you ever used that Cameron or do you know where that is in the classroom no okay well I just wanted to give that option to students because it is something you get included with your tuition. It's five hours per semester. So you can use that. I know sometimes dissertation students use it or students use it sometimes for stats. So you would not be alone in using it. But what I've heard is that you're supposed to book 48 hours in advance, but I've also been told by students it doesn't take that long. So, you know, try to, if you're going to use it, try to book it at least a day in advance, I would say, so that you can ensure you get a match. But, again, after we go through this, I'll try to maybe look in the class and see if I can show you where the tutor.com link is. What you should do is if you're using it for statistics, say that your class is math and that the type of tutor you want is or the type of class is statistics. You could say introductory or intermediate statistics and you would get matched. Obviously, I just say that because if you say psychology, there's not an option for like program evaluation for a class or whatnot. So I would just specify that your class is math and statistics so that you could get the correct kind of tutor. Okay. So let's go in and I'm going to, let me see how I can, let me see if I can, what I can share here. I don't know why my particular screen is not showing up. Okay. All right. Bear with me. We're going to get this going here. Okay. So what we can do when we're creating a data set, I do have an example in the classroom of the video I created for a different class where you do data entry and SPSS and then I also put a Jamovi one in there but what we're gonna want to do is we're gonna have participant numbers so let's say my participant numbers I'm gonna have 10 participants I'm gonna to just say 101, 102, 103, 104, 105, 107, 108, 109, 110. All right. And then I want to, um, create my pre-test scores for perceived stress. Um, and I might have some people who have very low, you know, pre-test scores are kind of reasonable. I might have some people who have very high, you know, scores. I'm going to put, you know, some different data in here to indicate some different perceived stress scores. Now, before I get too far into this, I do want to set up this. So instead of calling it B, I'm going to call it PSS-PRE. And I want to know what kind of measurement type this is. This is reviewed in the videos as well, or at least it was in mine. And this is a continuous measure for perceived stress. So I don't have labels because there's no categories. People could just fall anywhere between zero and 40 for that data. And I would do the same thing for my post-test score. That's also going to be continuous. And then for my ID number, I'm going to just, that's also going to be, uh it's just a measurement type it's just id so it's not a real variable um so it's just to kind of keep track of things okay so now my perceived test score my perceived stress i'm going to see did my perceived stress values go down between pre-test and post-test well some you know maybe sometimes it does, sometimes it doesn't, right? So I'm going to put some different scores in here and then we'll see, you know, what some reasonable values, but you know, we would have a range of possible values and then we would see, you know, we would analyze this to see if there's actually a difference between pre-test and post-test okay so that's it's pretty simple and straightforward you need to set up the data and then I want you to take a screenshot of this I am going to save my data file I'm just going to call it week five. And then I would be doing some descriptive statistics with this. So I would then go to exploration and descriptives, and I would be able to get some descriptive analyses of these so I can see what the mean is for each the pre and the post it was 24.8 and 22.8 standard deviation is 10 and 8.7 and I could also do a plot like a simple histogram or I can see I can check this out and look like is this normally distributed It does not look normally distributed to me, but I could just kind of, you know, get a sense of what my data looks like. I could also get a box plot so I can kind of more easily see if there's outliers. It doesn't really look like there's outliers, but that's just an example of, you know, know, some of the, you know, how to set up the data and how to, you know, do some descriptive statistics. Any questions at this point? We're just doing descriptives or did you want something about the change? Well, we're not really looking at the change yet. That's going to be next week okay we're not analyzing it yet so um yeah so we're not going to we're not going to look at that yet um i'm going to just show you one other thing to do with this data okay now I mentioned that this did not look normally distributed so when our data is normally distributed we like if this was a bell-shaped curve we would want to report mean and standard deviation. But because it is not, I would want to report median and interquartile range. So let me see. When I go back up here now, it'll show median. So the median is 23 and 21 .5, and the interquartile range is 16 and 15.5. So you could report mean and median and standard deviation and interquartile range because we're not really analyzing for whether it's normally distributed yet. But next week, you will have to be reporting whether it's normally distributed or not because that'll determine what kind of analysis you'll do to determine if there is a significant difference between the pre-tests and the post-test scores. Okay, so this week, what you need to report, let me pull back up the summary. So for your summary, you're going to report what your descriptive statistics say about your hypothetical sample, any unusual values or missing data. I don't have any missing data. And I don't think there were any extreme outliers like looking at those box plots. But I could say, well, there was a small decrease between pre-test and post -test based on looking at the, you know, means and standard deviations. Or if I was going to be more strict and say this is not normally distributed, looking at the median and interquartile range, there was a small decrease. But I don't know if we're not going to know if it's significant until we do our statistical analyses next week. But I could just say it looks like there's a small decrease. So maybe which indicates there may be, you know, slight improvement or something like that in stress scores. OK, so hopefully that'll be pretty straightforward. Any other questions before we wrap up? I guess I'd just ask if you're doing anything for Thanksgiving. Is family coming in? Well, my older son's home for college, so that's very exciting. And obviously, my younger son is at home, so he's very excited to have his brother home. and um my parents were supposed to come up but unfortunately my dad has to have a procedure tomorrow I think you were in my class when he was in the hospital for a few weeks so unfortunately he has to have some something else done you know he's just having some heart and lung issues so I think we're going to go visit them hopefully it'll work for us to visit them they just live like two hours away um otherwise I guess we'll just have you know a little holiday at home, but I'm hoping we can go see my parents. I mean, I hope my dad's well enough to be up for it. How about you? Do you have any? No, nothing so exciting. Just me and then the museum director and the curator. Oh, we do have a visitor who visited this week. She's going to be there oh nice nice she's working on her doctorate also and but hers is in in december and she's very stressed oh i bet and is really happy that she found a place to go for thanksgiving and get away from it for a little while well that's good oh i hope you have a very happy thanksgiving um that's nice that'll be nice that you that you guys get to have a celebration well hope you have a great week and happy Thanksgiving. The other thing I just want to mention before we wrap up is that I do have the remaining weeks of the course open now. Now that you guys have your measures and everything's kind of more streamlined going forward, like you guys have kind of gotten through in some ways the hardest part, but I know you still have analysis to do. I wanted to open up the rest of it just so you guys can see what's coming ahead. So if you want to take a look at anything, you're welcome to do so. And we'll meet again next week on Tuesday. I think it's at six o'clock and we can go through the next, the remaining steps for your report. But you guys are doing a great job and it's all really coming together. So hopefully you can enjoy the holiday weekend. all right thanks dr kelly thank you take care bye bye