Psychology week 7 assignment

program evaluation. This is our last Kaiser Live. Next week we only have a discussion and we'll be wrapping up before the three -week break. So next week just be sure to get your discussion posts all you know your initial post in Wednesday and your responses in by Wednesday I mean Friday. So this is the last big push. So we've been working all term towards a longer assignment. So I'm going to talk about that where you'll need to review what you were asked to do before in weeks one to six and revise that and bring it together into a new bigger report and then add on a discussion or conclusion section, you know, where you'll talk about things like were your outcomes met or not met did you run into any unexpected issues what kind of recommendations are you going to make for the agency so um that's you know you're going to bring it all together so obviously make making incorporating the feedback and making changes is really important um you're going to lose a lot of points if you just submit the same thing The maximum points you can get on the assignment is a 70 if you don't make revisions from the prior weeks, because this really is something where you need to, you know, update and make changes. But I'll go through all of that shortly. So to get started, this week we have a discussion That's optional, but I just want to mention it here in case you guys have anything you want to talk about during our session tonight, because this is really more of a feedback discussion. So I'm giving you time to, you know, and space to work on your final project, but I would like you to just use the discussion this week to clarify any issues or questions about the program evaluation. You know, what if scenarios, anything else that would help you? And obviously, just reflecting on the fact that we can only scratch the surface of program evaluation at eight weeks. So let me ask you guys, how do you feel about program evaluation at this point in the course? Like, do you feel like you've learned some new skills? Do you think these are things you could use in the future? I would just love to hear some feedback from you guys. Yes, I think that I have learned a lot because at the beginning, as I said, it was a little bit confusing to me. and I didn't understand where we were going. And, you know, I feel, you know, everything has become much more clear to me right now, even though we still learn the basics. I think that I would need to read a lot to deepen my understanding of what programming collaboration really is and what it requires, but I think that I have learned and developed a lot of skills, new skills. Okay, great. Well, that's very helpful feedback. Yeah, I think that there's really moving from a researcher mindset into an applied evaluator mindset over a relatively quick period of time. And these are skills that you could use working in program evaluation. Obviously, you wouldn't be hired right out the gate to conduct a program evaluation, but you might be able to work on one. you know, in some capacity, helping with projects and learning additional things about the different aspects of program evaluation, kind of on the job, which I think is a lot of times one of the best ways to learn. So let me ask you, Watchner, I know you already posted to the discussion, so thank you for that. But when you say you were confused in the beginning, which I'm sure you're not the only one, do you think that had to do with thinking of it as like a researcher or you said you didn't really know where you were going do you think yes um there's based on the the course label i i saw i i view it as research i thought it was part of you know one of our research courses and um I think it was during the first or second class, you were, you know, trying to differentiate research. That was like, you've been trying to differentiate research from program evaluation. and you know that point I said to myself what is it exactly and when I tried to review the book I see everything just like research we have design we have research question we have all this stuff and for me it was a little bit hard to you know differentiate program evaluation from research. And I think that was the confusing print about it. But at the end of the course, I began better understand, particularly after the method section part, I felt that I am much more comfortable um when it comes to um you know different differentiate um research and um program evaluation and it was a little bit challenging to me as well um particularly when i had to um select my design because with program evaluation you can like um you can have groups particularly if you are um for example if you you are choosing a um quasi-experimental design but you can't like um um assign people to to to group the same way right that might not be ethical to do in this if you're giving some kind of beneficial treatment or something like that not only yeah but if you have a like people um on a reading list with the same characteristic with people that are um um you know that are in the in the program you can use these people as a control group um yeah to compare and i yeah that it was about it yeah no that's a good point because that is something that's quite different from research because in research the gold standard is you would randomly assign people to an experimental or control group and have like a pre-test and a post-test. So in a program evaluation, we do have a pre-test and a post-test, but we don't have that control group because there could be ethical issues with that, unless we were doing something like you said, like having a wait list or, you know, maybe a group receives some different type of intervention that we assume is you know also beneficial but it's not it's not it's not the same thing as um the experimental control group design so thank you for clarifying that that's helpful to hear about. Yeah, I read your discussion post and agree that was a pretty good insight about the differentiating research question versus a program evaluation. And yeah, the waiting list is an idea for experimental design, but I appreciated in this course that we really just stuck with the pre post design and that way we could instead of focusing on the maybe the ethical consideration or the what did we choose and all that we're all speaking the same language and I've appreciated getting to learn in this course about what is the expected language to use. Great. Yeah, because that's definitely a difference in the report. So I feel like it was probably a surprise that sometimes it was like, oh, you only need a sentence here, right? You don't need like this full flushed out like justification. Whereas in research, we always want to justify everything and back everything up and have lots of citations and stuff like that. Whereas because we're dealing with an agency and we want the report to be readable for stakeholders, you know, like the grant agency or whatnot, we want to keep it pretty brief and, you know, be able to really keep their attention and focus on, you know, what's the intended messages just make it really clear and concise so i think the part of that is that they already at this uh whatever agency you chose they already have the program they know they like it already and so we don't need to really explain too much why it's a good one that's true yeah it's not like we need to justify like hey you don't need to like beat it over the head you know but yeah Yeah, I think that the intro definitely is important and kind of like contextualizing it a little bit. But after that, it's like, yeah, we don't need all these big lit reviews or anything like that. Great. Well, thank you guys for sharing. That's very helpful. All right. So why don't we go into talking about the assignment? it. So your final program evaluation report is worth around a third of your grade. And so this really gives you an opportunity. If you did well on the previous assignments, great. And if you didn't, you can still improve because you have, you know, the opportunity to incorporate feedback. So your final report is going to include the sections that were submitted in previous weeks, as well as a new discussion conclusion section that will include the implications of the results, whether your outcome was met or not met, unexpected events and recommendations. So I want to refer you guys to Chapter 9 in the textbook because that's a useful place to get additional guidance in completing the final report. But the one big thing I really want to stress is that it's really important to address any feedback received on the section submitted in previous weeks. I take a lot of time to give feedback, you know, and that's because I really want you to learn. I really want you guys to be able to learn and grow because obviously what's the point if you're going to, you know, I could, I suppose I could give everybody a hundred on everything, but then if it wasn't perfect, then what's, you know, there's always usually something to learn and grow. So they're probably, you know, with the final project, I'm imagining grades will be higher because if you really incorporate feedback and you're consistent throughout, then you have a good chance of getting a good grade. Okay, so your final report should include a compilation of weeks two through six. So that's your weeks two, three, four, and six assignments. You're going to be adding your discussions and conclusions this week, and then you'll also add your logic model to the appendix. Okay, so I'm just going to review this. I've been doing this a lot of the weeks that we've met, but I'm just going to review. You're going to talk about the agency and the program within the agency that you used, and your introductory section goes over the problem, the program, and the evaluation. So there's two things I want to emphasize about these past sections. One is that you need to go back and make sure you're being consistent so if if you changed your outcome or like some people had an outcome that wasn't measurable with a standardized tool say for example something like number of doctor visits so if you had started out with an outcome like that and you just you know realized you had to change, then you need to go back and make sure that's consistent throughout your whole report. Another thing, so, you know, when I, I guess I didn't tell you how much, I guess I did tell you how much I'm cutting it off. So if you don't make any changes, the max points you're going to get is 70%. So let's think about that, like 10% being cut off for not being consistent. Another 10%, like if you're not consistent throughout your whole report about like what your outcome is and what your program is and what you're measuring, that type of thing. Another 10% is if you're not making revisions to past tense, right? So when you wrote this as a proposal, you said, I'm going to do this. But now we would write it in that just change it to past tense when that's appropriate, because this is something that you did, not something you're proposing to do. And then, you know, another 10% is really for, but I really think it's probably more than 10%, is really for the feedback, like that I gave you. But, you know, I gave a lot of feedback. So really make sure you're making changes, you know, because I try to give you helpful tips in writing a program evaluation report. Okay, so this is the introduction section. I feel like people did pretty well on this overall, so I don't have a lot more to say about this. The method section, this first part, the program definitions and design, I feel like that there were some challenges with that in week three, but then you had the opportunity to revise that and include that as part of week four. So I feel like you've already gotten some feedback on that. And so hopefully that's in good shape. And then in week four, you added your settings, participants, outcomes and measures, and data analysis. And if you'll recall that outcomes and measures sections was worth the most points. Why is that? Well, you know, we really need to know what we're measuring and how we're measuring it. So that is an important section. So just make sure that part is up to date. The results section should directly answer the evaluation question using the week six data set. Now, I haven't given you guys feedback yet on the week six assignment, but you do have feedback on weeks one through four. So you can certainly get started. And then I will try to get feedback on the week six assignment over the next day or two. But that should be pretty straightforward, incorporating your results section. and then now this is the new piece the new piece you'll be adding is your discussion and conclusion section so these are some of the questions that you should address the first question which is pretty obvious which would always follow a result section is did your outcome show meaningful improvement so not only statistical significance but practical significance does this Like if you look at the means and standard deviations or the medians and standard error, does this look like a meaningful change from pre-test to post-test? And is it supported with statistics? Does it make sense? Did things go up when they were supposed to go down? You know, what did you find and was it meaningful? Next, you should discuss any unexpected issues or challenges. Some of that can be a little bit hypothetical because this is a hypothetical evaluation. But overall, you should be discussing the implications for the agency. Was the program effective? Was it promising but not really strong results, maybe needing some refinement? Was it inconclusive? Like, what did you find? And then what do you think the agency should do? Should they continue the program as is or should they modify parts? Do you think there are certain recommendations you can do? You should have at least two recommendations. I always think that's really a fun part of the report is to try to, like, use your critical thinking to try to take the results and then think about, like, what would I recommend that they do with this? So are there, like, improvements they could make to the program or are there changes you would make to the evaluation design or the measures that you use? Do you think that the staff needs more training or do we need to do more monitoring of whether people are really doing it? That would be fidelity. Like, are they really showing up to these different trainings or whatever it is? And then finally, you would have a takeaway that would be one sentence about the program effectiveness. So, you know, general recommendations. So recommendations either way, whether you found what you wanted or you didn't find what you want, but it's important to include these different components in a program evaluation report. Okay, so this doesn't, again, we're trying to keep this brief. So even though I put a lot of points here on the discussion conclusion slide, you know, this doesn't need to be incredibly long. Still keep in mind and balance it with the rest of your report. OK, then for your appendix, you should include your logic model in the appendix. And then finally, I just want to add this grading note. So if my prior feedback is not clearly incorporated and or internal consistencies remain, like what your outcome is or, you know, other changes, if it's not consistent throughout your report, then the maximum possible score for the week seven report will be capped at 70 percent. even if you wrote a really strong discussion section, because it is worth a lot of your grade. I've had students ask me if they could revise certain weeks, and I've said no, because this is your chance to revise. It's worth, you know, like 30% of your grade, this final project, and you're revising for this week. And that's really an important skill too, whether you go into program evaluation or research or whatever else. Certainly in dissertation, you'll get feedback from your chair and your committee members, and you'll have to make revisions. So it's just kind of good practice to be able to take feedback that you've been given and implement those changes. So any questions before we wrap up? Do you have a sample available, or are we keeping it like a research paper, like introduction, method, results, and instructions? You should be keeping exactly what we have here. This is your week two assignment, introduction section, with these three subheadings. Then the next section is your week three and four assignment, but it should have the heading method, and then it would have these subheadings, program, definition, design, setting, participants, outcomes, and measures, and data analysis. And then the next section of the report is your week six assignment, so it's called results, and then you should just have your results in there with a figure. And then the last section, I would call this conclusion. I've kind of switched back and forth between calling it discussion and conclusion. But I think, you know, for program evaluation, it's good to call it conclusion. And then you can have some subheadings if you want, like recommendations or implications or whatnot. But you don't have to have subheadings in this section if you don't want. Does that make sense, Wachner? Yes, thank you. I remember you said before, unlike a research paper for the program evaluation, for the report, you do write introduction. Okay. Yeah, so this is what I mean by introduction. Your introduction is just this, what you did in week two But again, you need to update it If there's any inconsistencies with the later weeks Everything needs to tie together No, what I mean is Do we have to write the word introduction? Oh, I get what you're saying now Sorry, I didn't understand. Yeah, I think that you can write the word. Yes, you can write the word introduction. So like in a research paper, that's not APA format. You don't write the word introduction, but here you should. Okay, thank you. You're welcome. Okay. Any other questions or concerns. Well, thank you guys for coming. It's nice that you came. And I guess just as a reminder, there is extra credit for attending or watching the live session. So for those people who are watching the recording, be sure to fill out the survey. I have a survey each week that people can complete if they want extra credit. but thank you guys for coming and if you have any questions throughout the week feel free to reach out to me send me an email or you can post in the week 7 discussion area any questions you may have yes thank you one last question do we have any illustrations regarding the number of pages no but if i gave you feedback before like cutting lots of parts please cut those parts this is not supposed to be a 20 page report or anything like that right it's just it's supposed to be pretty like to the point um so yeah i think that's that's something that um is definitely different from other classes so um great well you guys are doing great and um you know i'll look forward to reading your reports um watchner are you i'm going to just turn off the recording now because i'm going to ask i will ask you guys questions that are more specific to your particular evaluation.