Psychology week 3 assignment methods

Hello, and welcome to week three of RSM 816 program evaluation. I'm recording this session because the university is closed in recognition of Veterans Day, and we want to thank everyone for their service, those who have served or the family of those who've served. So thank you to everyone who has served, and I hope you have a good day off. This is week three. This week, we move from what outcomes we are evaluating to how we will be evaluating our outcomes. We're beginning the methods section of your program evaluation report, and we will keep it very short and straightforward this week. So I'm happy to say that this week's work is among the most brief that you will have, not only in this class, but possibly in graduate school. So it's a short assignment. I do want to prep you, though, that although this is a lighter week in terms of demands, demands. We do have data entry and data analysis coming up in weeks five and six. So I want you to be prepared for that because that's going to be a lot of work. Week five in particular may be a bit challenging because you're setting up a data file that does occur during Thanksgiving week. So because it's Thanksgiving week, I've changed it to only a discussion that is due a bit later and no assignment that week. But it is still quite a bit of work, so I just want to give you a heads up that your discussion post will involve creating a data file. I'll give you instructions that week, but that is going to be a lot of work, so I just want to let you know. Okay, so let's get started and talk about what we're doing this week. For your discussion, we're going to be talking about the method section of your report and thinking through questions around your methodology. So you'll write what you would write in a report to the board of directors, the agency that you're working for. Or it's going to be just like a typical report that you're building towards for your week seven assignment. And it will look a lot like a method section for an APA paper, but it'll be pretty short. So you'll start out with the word method centered in bold, and then you'll have your headers left justified. So but your discussion this week is going to help prepare you for writing that first part of your method section. And so this is going to be you telling readers what the work is that's going to be done at the agency. So now that you've suggested one measurable outcome for your logic model, this week we're going to determine how the evaluation will be conducted and who will be included in it. So remember that even if you had more than one outcome, I want you to just focus on one outcome for these remaining weeks. So at the end of week two, hopefully you had one main outcome that you were talking about in your introduction. But if not, you'll go back and revise that. You can mention a secondary outcome, but we're really just going to focus on one outcome. Um, so last week we talked about the evidence -based research and statistics in terms of the problem and what kind of things agencies like the one you're working with have been able to do. And now we're going to talk specifically about your program, um, what they're doing, what's going to be looked at in this evaluation, and its particular results. So as the slide outlines, the discussion this week focuses on four things. The first of the four is the outcome being evaluated. So that's what you selected from your logic model. Obviously, there were short-term, intermediate, and long-term outcomes, But now we're going to just focus in on one short or intermediate outcome. And remember that from this point onward, your discussion should always be about your specific program. So make sure you're not just making generic statements. They need to be specific about your program. Okay, so for the outcome being evaluated, you should simply restate one outcome from your logic model, not the goal or not the activities, but really what the outcome is that you're working towards. For example, client depressive symptoms significantly decrease from time of intake to post-participation. So not CBT sessions are provided weekly or anything about what you're doing. Um, we just want your outcome to be focused on what you're evaluating and the change should be something measurable, not whether or not a service actually happened. Um, and so that's, that's really primarily what you should be focused on with your respect to your outcome. Okay. Now, I'd like to spend a little bit of time talking about your evaluation design. So in program evaluation, we almost always use a self -control pre-post design. And that means we compare each person to themselves, their intake score compared to the discharge or end of service score. So we don't manipulate anything as researchers, which might differ from what you do for your dissertation. But in a program evaluation, we don't add a control group or we don't add randomization. We don't change anything at the agency. Now, if the agency has naturally occurring different treatment tracks, you can mention those, but we do not create groups just to have a comparison. So you would not be mentioning any kind of other groups, you know, because, again, the agency is the one who designs what's actually done in terms of their intervention or treatment as clinicians or therapists. And we're just there to evaluate how it's going, not telling them what should be done. So in terms of the design, really what we're looking at is intake data versus outtake data. For example, is there a decrease in symptoms or is there an increase in quality of living? um so what we're looking for is is there a difference between the intake test and then the end of the period test or post-treatment tests um and as a result most of us or most of you are going to be looking at a pre-post design um so most program evaluations do involve a self-controlled pre-post evaluation, which is the minimum design that's typical for this class. It's a typical, you know, we only have an eight-week class, and that's typically what's done at an agency. Now, be sure to look at your book and talk about other designs when you answer this. and um if your agency talks about or i mean not the agency the program that you're dealing with talks about different in service groups you can talk about those um like for example maybe your your program does cognitive behavioral therapy or motivational interviewing and then or maybe they look at dialectical therapy. You could look at between groups for those as well as the pre-posts, but only if that's something that they specifically talk about and that you found information about. So my guess is that because not all of this information is available on the website that most of you would just be doing a pre post design. But I want you to, you know, just make sure that you're thinking about these issues. So for example, if you're doing foster therapy, foster care, and you're you're trying to look at a program and see whether there's improvements from pre-test to post-test with foster care, you'd report what kind of things they're doing in between, like maybe at the foster care agency, they're doing family therapy and individual therapy, but not everybody who's in foster care has a family who's willing or able to participate in the family therapy. So perhaps some people would have individual therapy only, and other people would have individual therapy and family therapy. So then you'd want to look at those differences. But again, we would not be creating any group like that. That would only be if it was a naturally occurring difference. Instead, we would typically say, like, maybe everybody has cognitive behavioral therapy and stress reduction. And we would just take a look at what the scores were of people before, starting the program and after ending the program. So the key is that you're not setting up any groups. Agencies set up and do the work in terms of the program and the implementation of the program. So we're just going in to say, how's it going, Matt, what you should do. And so we're looking at is there any difference pre-test to post-test so we're just um we're looking at whether there's any difference so I want you to look at the chapter in your book and um you know really evaluate based on your program that you're looking at is the pre-post design okay does that work? If not, why or why not? And again, don't create any new groups. So sometimes there's naturally occurring groups like a wait list. So those would be people with the same symptoms, but they just couldn't get into the program because it was full. But we wouldn't want to create a wait list because that would be unethical. So we're not. So I would say in general, in this class, I'm going to assume that most people are going to be just a within-subject design, pre -test, post-test, but I want you to really look at this and determine, based on your reading and your reading of the program, whether that works for your particular program evaluation. So you're going to be building this, you know, the method based on, off of your outcome that's from your logic model your evaluation design that's going to be based off what the agency's doing and you're reading in chapter three um and and um you know but i want you to also discuss um limitations of the design because we're not using a control group there will always be at least one threat to validity. So I want you to pick one of these limitations and explain it in normal language in terms of what the limitation is, you know, for your particular sample. So it could be something like maturation, where participants improve over time, even without treatment. It could be something like selection bias, where the people who seek treatment are more motivated to change. It could be history effects, people improve over time, even without treatment. It could be regression to the mean. Whichever you choose, keep it simple and tied to your outcome. And then last but not least, I would like you to talk about your sampling strategy. Again, really look at chapter three, and I want you then to describe who will be included in the evaluation. In most cases, your sample will be clients who receive a program during a specific time frame, like clients admitted between January and June, 2026. And I want you to explain why that sampling approach is appropriate for your program. But generally, I want you to, you know, Most program evaluations use everyone who received the program because, again, we're not going to do random sampling. We're not going to just choose only certain people. We're going to want to include everyone if possible, especially because the size of most of the agencies you're working with are small enough that you should be able to sample the entire population. So you'd sample those people who are intaked. That's a strange word, but that's a word we used in clinical and evaluation work. And so your population is your sample. There's no assignment to groups, no random sampling. And the population would be between particular intake dates. And it would be rare to do something different. If it's a very big agency, you might want a smaller time period, like January 1 to January 30th, 2026, instead of having a six-month time frame, have like a one-month time frame. And if you're using sample and not looking at treatment outcomes, but instead at something like housing outcomes, you know, you want to think about what measure you're looking at and whether time of year would impact your sample. For example, let's say we want to look at whether housing outcomes improve for a homeless population. We might have really different results or intakes in winter than in summer and vice versa in hot climates. So being near Chicago myself, we would have a lot more intakes in winter than in summer. so in that case you might want to have like two two or three months in the winter and two and three months in the summer so that you get a better selection of people who you include in your sample but the main thing we're looking at with an evaluation like this is in -tech data, intake data versus outtake data. And again, we want to try to include as many people as possible, but make sure you're doing your readings and in particular chapter three this week and figuring out what kind of sampling you would want to be doing. So feel free to reach out if you have questions about that as you get started before you post to your discussion this week. All right. Now, in terms of your assignment this week, last week you worked on your intro. This week you're going to be building the method based off of what the agency is doing. And after you work out in the discussion some additional things about your method, you'll be writing your method section. In particular, we're going to talk about three parts, the program, the definitions, and the design. And then you'll work on the rest of the method section in future weeks. So again, this is a shorter assignment, but I want to remind you week five is a lot of work. So just be prepared, although I did try to simplify it for Thanksgiving week. Okay, so first off, we have the program section of your report, and this whole assignment should be less than one page. It may not even be more than a paragraph or two. You're not writing analysis. You're not selecting measures. You're not entering data yet, so it should be pretty brief. Okay, so in terms of the program, I want you to be talking about what the program is looking at, and so describe how the program operates during that evaluation period, what services are provided, and how they're delivered, and I want you to talk about who delivers the services. if you're able to determine that, what happens in the program, and how often clients typically participate. And if it's unknown, make a reasonable assumption and look at where the program takes place. So for example, the program will evaluate whether the program significantly reduces fill -in-the-blank, like are they looking at whether it reduces stress or PTSD, substance *****, homelessness, whatever it is, and try to fill out the details as best you can from the information on the website. Okay, next, in terms of the definitions, here you should be defining the population of the program serves, the specific outcome you selected from your logic model to evaluate, and state that outcome. But don't select a measurement tool. So first off, I want to say about the definitions. Some of you may not even need this section because you may have already talked about it in your week one submission. But if you didn't, then you would need to describe this. For example, um, let's say your, your program does, um, cognitive behavioral therapy and motivational interviewing. Well, you would need to find those things and talk about, you know, what they're doing and what that looks like. Um, but, um, you know, if you haven't, if you haven't talked about that yet, then be sure to really describe what's being done. So, you know, and then make sure that in one of these sections, you have a statement such as clients experience a reduction in blank during program participation, or maybe it's something like this program shows a significant change and blank from pre -test to post-test. Okay, and then for your next section, the design, briefly describe the evaluation design that is going to be used to examine the change in your selected outcome. Okay, so this part you should have worked out in your discussion and this can even just be a sentence like this program evaluation will incorporate a self-contained pre post design to determine if significant results are realized between intake and post treatment okay so again the total assignment for this week can range from a paragraph to a page, but no more than one page. So in review, we're building our report. Last week, you worked on your introduction section review, talked about your program, what the problem was, the program itself, and the evaluation. This week, you're writing up the first part of the method, talking about the program, the definitions, and the design. And if you're looking for a sample be sure to go to chapter 11 I mean not chapter 11 chapter 9 in your book and that has some additional information for you so hopefully that's helpful but please feel free to reach out during the week if you have any questions and don't forget to complete the extra credit survey so you can get credit for watching this session thanks and I hope you all have a great week. Bye-bye.