Psychology WEEK ONE ASSIGNMENT

Hi, Wachner. Welcome to class. Thank you, Dr. Kelly. How are you? Good. How are you? Good. Thank you. Great. Nice. All right. Well, I'm looking forward to getting started on this class. We are getting started on program evaluation. And some of you might wonder, well, what is program evaluation? See, why is this not moving? My slides are not moving. Okay, then I moved too much. All right, there we go. Yay. Okay, so program evaluation assesses the effectiveness of programs. so many times if you have a program if you want to get grant funding you need to be able to show that the program is actually effective um so this is actually a class or a type of training that's actually a great opportunity for phds and psych because we're not a clinical program program so many of you are not going to be working in the clinical setting for the most part unless you already are with some other degree you have but for those of us who like research and statistics and analysis program evaluation is a great and there are jobs there are job opportunities with this where you can evaluate the effectiveness of a program or an agency. So it's a potentially good job opportunity or way to get out there and get a job with your credentials after you graduate from Kaiser. There are some differences between research and program evaluation. So for example, program evaluation cannot change the programs, only evaluate them. We can look at what they offer now and evaluate the outcomes. But unlike an experiment, we can't randomize treatments or programs. And even though we can't randomize, we can do, we can follow like research expectations. But one thing we won't do that follows research expectations is informed consent. So typically when we do a research study, we have an informed consent before we start the study. But as program evaluators, we don't have that because people who participate in the programs agree at the beginning to take part into that agency or program, and they already agree to complete consent. questionnaires at the beginning and the end of the program. So we don't have the traditional informed consent that we have with a research study. So unless you're going to publish, but that would be different. That's not typically a main goal of program evaluation. so um so why do we do program evaluation um well i'm going to get to introduction next so we want to make sure the program's worthwhile and grant agencies typically want to know that it's beneficial and so that they you can determine you may be able to determine that in the future some changes may be made and then maybe you can apply for a different grant given certain changes but typically we're just looking at what they're doing and is it effective and that that can be used for to help them get funding or to support the funding they already have. So we're going to in this class focus on some programs that are already up and running and not for profit and the reason for that is we are not interested in whether these programs are effective in terms of making money. Instead, we're interested in whether they are reaching program goals. So for example, maybe the outcomes that they're looking for are having better educational attainment for participants or some kind of symptom reduction, or maybe it's more involvement with family or reduced, you know, addiction. So there are a variety of different outcomes that could be looked at, but, you know, profitability is not one of them. Okay. So to get started, why don't we go around and each tell each other something about yourself. I'll start. My name is Dr. Kelly Schmidt. I have had some of you in class before. I am a professor at Kaiser and I have been here, I think, um almost 13 years and i have two kids and i can kind of remember how long i've been here because i had one of these kids the first year i was here so he's 12 12 and a little bit right now and my other son just started college so he's in his first semester of college so who wants to go next. Cameron, you have your camera on. Would you like to go next? Just introduce yourself. I'm Cameron Cook, and I live in Seattle. I've lived in the same neighborhood for 15 years now, and it's nice being here and being part of the Unity Museum, where we, it's a free museum, So we get all kinds of people and it's a very colorful street. So as a majoring in psychology, it's kind of like ideal because of all the different kinds of people that we get to meet. Nice. Great. Thanks. Welcome to class, Cameron. Um, Watchner, I heard you had your microphone on before. Would you like to go next? Or Lau? I know Lau. I don't know if you have your microphone on, but okay. So Lau says I have two kids. One passed away 11 years ago. Oh, I'm sorry. And the other son is 31. PhD is a dream. I'm at work. I can't talk. Oh, well, we look forward to helping you reach your goals. That's a good accomplishment. Thank you. Yes, you were just in psychometrics as well as Cameron. So welcome. Welcome back to class. I look forward to working with you again. All right. So I don't know if Wachner has his microphone or the chat on but yeah yeah hello yeah his mic is working so we can hear you okay go ahead Wachner there's a feedback I don't know Okay. Okay. My name is Wachner Lazor. I am originally from Haiti, currently living in Orlando, Florida. What else? I have two kids as well, a boy and a girl. um they are teenagers now and um as for the program i would say um it's a it's a life dream and i am really interested in teaching and for me um earning my phd degree would um allow me to um move further with my dreams and um starting teaching i am very um interested in um cognitive neuroscience so um it's basically about it great well that's very interesting well welcome welcome to class watchner i don't think i've had you in class before so i look forward to interacting with you no i just met you um during residency ah that's right great well welcome thank you all right um okay well i'm going to just turn my camera off because i'm getting a little bit of background noise too so um i just want to mention i don't know if any of you did the back baccalaureate entry or the master's degree at Kaiser. But there are some similarities between Capstone and program evaluation because Capstone, those are the last two classes in the master's program, do a type of evaluation. But a difference with RSM 816 is that we are using an actual agency. but and of course this class is only eight weeks whereas capstone was 16 weeks but you know we can choose an agency that's not just limited to mental health or substance ***** but we you know are giving you a selection of a few agencies to focus on just to kind of try to help help and make it more smooth because it is only eight weeks. If you had to go out and find an agency on your own, that could add some extra time and challenges as well. Okay, so let's get started and talk about the discussion. So all of the discussions and written assignments in this course will focus on a program evaluation of one specific nonprofit agency that you're going to be choosing this week. So each week's discussion thread will focus on the issues that will be relevant to that week's written assignment, and you should use the discussion as a means to clarify your understanding of the issues related to the topic for that particular week. So this week, you're going to select an agency, get information about the agency, and develop a logic model that provides a graphic overview of the program's purpose, services, inputs, activities, outputs, and outcomes. Okay, so that's just kind of a starting point. And I'm going to go over the logic model, so don't worry. I'll go over that with you. For the discussion, I want you to choose one of three agencies. There's TIDES Family Services, National Youth Advocate Program in Florida, and the Grand Concourse Counseling Center. So you can choose any one of those three, the URLs are in the classroom, and so you can explore them. So explain why you selected that agency and what drew your interest in it. Next, I'd like you to look at the agency's website or whatever materials you can find about it and talk about what population it serves. Like, is it serving kids? Is it serving adults? Is it serving, you know, people who have trauma or addiction? Like what's the specific population that they're serving? What services or interventions are they offering? And what are the stated goals or intended outcomes? So that last bullet, like what are the stated goals or intended outcomes is going to be really important because that's That's what, you know, you're really going to have, you're going to have to choose one outcome to focus on for this class. So, you know, you can look. Sometimes agencies have more than one outcome. And so you can kind of check them out and see if there's a particular outcome you're interested in. So next, the next part of your discussion, and each of these should be like choosing the agency and why you selected it. That could just be like two sentences. The program overview, I would say that should probably be about a paragraph. The information gathering for your logic model, I just want you to start gathering some information for your logic model. So this is not fully flushed out, but just to kind of start looking at the logic model in your textbook and start thinking about some of these things, just kind of have a little bit of a, like a little bit of an outline. And then ethical considerations, that would be another paragraph where you would discuss how client confidential, confidentiality could be maintained. So that's kind of hypothetical because you're not actually going to be interviewing the clients. But say, like if you were collecting information with a therapist or some other means, like you'd have to watch for breaches of confidentiality. So how do you make sure that the information state stays confidential? So this would be kind of more hypothetical if it's because I don't expect you to have that information like available on the agency website. But like think about what kind of agents, what kind of ethical practices are good practices in evaluation. So for example, you know with the program evaluation hopefully now you know like people are interviewed at the beginning and at the end so like intake and discharge scores how would you keep their information confidential like maybe you assign them an id number but because you have to have their information at the beginning and at the end you'll want to make sure that the names are stored somewhere separately, like the names and ID numbers. So maybe you keep them somewhere separate, locked up and secure. But, you know, when you're doing the interviews, it would just be ID numbers. So you're just going to be working with ID numbers, but just, I just want to make sure you're thinking about those issues. And are there other potential conflicts of interest or other ethical concerns to consider. Like, for example, with an evaluation, we wouldn't release information about a person unless they're a danger to themselves or others, because if they were, you'd have to report it to the agency. But that information would not be included in a program evaluation. It's just that would be, you know, something that you would have to do, you know, ethically just to make sure that the person's not at harm. So when you're actually working on your actual program evaluation, which is you're going to come up with a big program evaluation by the end of week seven, you may be happy to learn that you won't be writing a lit review as you do for some classes. This is not a traditional paper, and you're going to do each part of the paper each week. So each week you'll do a different part of the paper and then you'll incorporate that feedback and then submit a revised paper in week six with all the different parts. But it's going to be a lot of bullets, not necessarily full sentences or APA style. So if you're interested or curious, you can look in week two. There is an example of a program evaluation report, although that's a PowerPoint, it'll give you an idea of what a program evaluation looks like. So you're not actually carrying out the evaluation, but when you write these things up, I want you to write it up as if you would be doing it yourself. How are you going to maintain client confidentiality, things like that. Okay, any questions about the discussion before we move on to the assignment overview? Okay. Hi, Katrina. Thank you for coming to class. Hi. Hi, Katrina. Um, so, um, okay. So for the assignment, um, you will, there is a logic model template in the classroom that I want you to download and you're going to complete that logic model and use the template just to describe your program's inputs. That's the resources, activities, what the program's actually doing, the outputs, the measurable products of the activities, like how many clients they serve, how many sessions people have to attend, or how many referrals they get. And then the outcomes, those are broken up into three types of outcomes, short-term, intermediate, and long-term. And so those are really, you know, what hopefully the program does and, you know, that it's effective at doing. So when you're thinking about this, think about, you know, how might you be looking at some of these outcomes? Like, let's say, you know, how would you be looking at right now? We're not going to focus on measurement yet, but if you want to like have participation, how might you look at that? Or if you want to look at trauma reduction, how might we be measuring that? Or alcohol addiction reduction, how might those things be examined? So you are going to be choosing one outcome from your agency to focus on. So that could be mental health. It could be trauma, job training, peer involvement, family functioning, you know, attending more housing workshops, whatever it is. You have to look at the agency information and find an outcome of interest to you. And when you're thinking about this, and again, we're not going to really focus on measurement this week, but when you're thinking about it, you're going to have to choose an outcome that's not just measured with a yes or no answer. So, for example, if you were to say, have you ever experienced depression, you know, that's very unlikely to have changed between the beginning, the intake and the outtake. So you wouldn't ask a question, a yes or no question, because, you know, have you ever experienced depression? The answer is going to be the same more than likely at the beginning and at the end of the program evaluation. So instead, you might say something like, have you ever experienced, in the last two weeks, to what degree have you experienced depression? You know, so that there's different answers at the beginning and at the end. So we'll talk more about measurement next week, but I just wanted to mention that. So this week for your assignment, you're going to submit two documents, the logic model, which is just, you know, filling in the template and then a one page narrative explaining the assumptions made, how your model aligns with your chosen outcome. You know, for example, anxiety, depression, whatever your outcome is and the information you wish you had but couldn't find. So, you know, obviously we're working with pre-existing agencies, so there's going to be some limitations in terms of what information is there and is not. So just a few tips, make sure you don't confuse outputs, which are what you do with outcomes, which is what we, you know, what change is. For your outcome, make sure it's not too vague, like better mental health. You know, you could be more specific and say something like lower anxiety or decreased depression or lower alcohol addiction or more attendance at medical visits, you know, whatever your outcome is, you know, and make sure you're using the template in the class. Okay, so I'm going to talk some more about the logic model, because this is what you're going to be filling out. So this is a screenshot from the book. But before I start to delve a little deeper into logic models, does anybody have any questions at this point? Okay, so by the end of tonight, I think you'll hopefully feel better about creating a logic model, which is due on Sunday. that's a one-page visual that explains how a program is supposed to work. So every program out there is doing something for a reason, but we want to write down those reasons because it would be really hard to evaluate it if there's no clear reasoning for what this program is trying to do. So a logic model helps us make the reasoning visible. Okay, so as you can see here, and I'm going to go break this down more clearly because it's a little hard. This will be a good resource for you to look at with the book, but it's a lot of information. Basically, it's breaking down like what resources do we have? I say we as if I'm part of the program, but what resources does the program have? What is the program doing? What is it producing? And how do people change? So kind of related to Cameron's joke there in the beginning about change, right? We're going to look at change here and see what changes over time. So the form you're going to complete this week is going to follow that structure. And so as we go through the example tonight, I want you to list on for the differences between outputs, which is what the program does or delivers, right? So I'm going to try to point to outputs here. So this is what the program does or delivers, and I want you to focus on the difference between that and the outcomes, which is what changes because of the outputs, right? So we're focused on the difference between outputs what it's doing and what what might change okay over time and remember we're looking at short-term intermediate and long-term change so there there could be different obviously there's often very different short -term versus intermediate versus long-term change okay so this is kind of like the form that I put in the classroom this one could be found online but I just kind of simplified it because you don't need this report part and you can just edit in the boxes or if you don't like the template that I put in the classroom I'm okay with you using a different template because there are a lot of them online as long as it has these same categories inputs outputs outcomes short medium term and long term OK, so you could Google logic model social services and you would find a lot of examples or you might even find some templates. So you're going to fill in your chart like this with bullet points. OK, and so what are the outcomes that we hope or the program would hope that would come out short term, medium term and long term? OK, so, for example, if it was a therapy program, you know, what are the outcomes they want? Or if it's a drug reduction program, what are the outcomes they're hoping for? So you would be writing out your model. And again, it's bullet points. OK, so I'm going to go through a little bit more about logic models just to kind of try to help you understand a little bit more about this, because I don't know. Have any of you ever heard of logic models before? No. Yeah, it's a bit different, right? It's not something that we talk about in, you know, probably the vast majority of your other psychology classes. Okay, so with a logic model, so first off, before an evaluation, we need a shared understanding of how a program works. So a logic model shows how the inputs, activities, and outcomes connect, and it bridges what a program does with what it hopes to achieve. Okay, so basically a logic model is a visual map of a program, and it shows the flow from resources to activities to expected results, and it keeps program design and evaluation aligned. So I mentioned a little bit about this already, but what are logic models used for? They can be used for planning, communication, and evaluation, and they clarify goals and identify performance measures, and they help staff and funders see the priorities clearly. So, you know, this is very useful not only for the program evaluators, but also, you know, internally to make sure everybody's on the same page. Simply put, a logic model is a chain of if-then relationships. So if we have the resources, then we can provide activities. And if we provide activities, then we expect change. So it maps how needs lead to results. Okay, so here are the logic model elements. So the inputs are the resources, right? So we have the inputs are the resources here. Those are the programmatic investments available to support the program. and then the activities are what the program does and the outputs are the product or service delivery implementation targets to aim to produce here we have the target population which is the users of the products or services and then of course we have the outcomes, which are the changes that occur. And this structure keeps the model measurable and objective and organized. Okay, so let me just go through a little example here. So here's a simple logic model. So in this little simple logic model, this example shows how the different elements fit together. So like, for example, here we have a training session is the input, the activities is attending the training session, and then the outputs is the notes and learning based on attending the presentation. And then the outcomes is, you know, what the, hopefully the deliverable is like increased understanding. Um, so, um, this is just a little example, um, of where outcomes equals actions, outputs equals actions, and outcomes equals effects. All right. So I'm going to go through a case study here, which is Wonderland County's teen mother prenatal care program. So the problem here is that in Wonderland County, teen moms are not delivering healthy babies, and the county health department identified lack of prenatal care as a risk among this population. They weren't going to the doctor enough, not getting enough prenatal care. So to address this issue, Wonderland County City Council released a request for proposals to fund programs that improve birth outcomes among teens. So that's very common. I don't know if you've seen requests for proposals, but there often are requests for proposals where people can get funding for a particular need. So in response to this RFP, the Orange Blossom Health Group plans to implement the Teen Mother Prenatal Care Program, which is a certified prenatal care education program. which is considered a best practice, and the groups preparing a logic model as part of their grant application because they want to get that funding, right? They really want to help educate teen moms on the importance of proper prenatal care and hopefully with a longer-term outcome of delivering healthy babies. Okay, so let's see. Here's an example of a logic model using this example. So this is a visual summary of the logic model. So up here in the left corner, it says the target population is pregnant teens between 13 and 18 years of age. The goal is to educate at least X percent of the pregnant teen population attending Wonderland County Public Schools about the importance of prenatal care and delivering healthy babies. So they would obviously fill in this X percent, but you would have some kind of measurable goal, like is it reaching the desired amount of teens? So the inputs is, you know, educational materials, the funding, the staff, the space for the prenatal health classes, and of course, pregnant teens who would be coming to it. The activities include teens recruited into the program, and then they would be asked to attend prenatal health classes. The outputs would be the number of pregnant teens who attend the program. And then the outcomes would be, again, there's three different outcomes. Short term, have teens be more knowledgeable about prenatal guidelines. Intermediate, have them follow the proper prenatal guidelines. And long term, have them deliver healthy babies. So sometimes these are kind of listed as more like, you know, you know, change in attitudes or change in behavior, whatnot. It could be various different things, but it is important to kind of just graph out, you know, short term, intermediate and long term. Um, so again, outputs are what the program does or delivers and outcomes are what changes because of it, because of the program. Okay. So, um, this logic model, um, is, you know, really just here to help us think about, um, you know, a structure, and I want you all to be working on building a logic model. You know, it could be something like the template we looked at, and the form you'll use for your assignment matches similar columns, so resources, activities, outputs, and outcomes, and, you know, you'll just take information about what the program already does. I don't want you to try to fix it or redesign it because we're not going to be able to change the program. Remember with program evaluation, we're not changing what they do. We're just trying to evaluate, like we're basically trying to give them some useful information so they can get funding and know is what we're doing really working. So that's what we'll be doing with this program evaluation. All right. Any questions or comments at this point? Okay. I know it's kind of a lot of information, but I promise that we will go through each week by week. So you kind of start off getting up and running right here with these logic models, but it's not too complicated. You can kind of keep it simple. Like, you know, there is a guide in the classroom that gives you some tips in terms of what you can look for and, you know, what kind of information you should be including. But obviously, there are some limitations because normally we would want to interview somebody from the agency, but there's just unfortunately not enough time to do everything in an eight-week class. So you'll just have to use the publicly available information to try to make sense of your program and what you could evaluate from them as best you can. Let me see if I can pull up that guide. So yeah, there is a guide in the classroom. So it says here, like, these are just examples of evaluation measures you could use, you know, that measure, like anxiety, depression, things like that. But like I said, we're not focused on measures yet. We're just more focused on outcomes, you know, but when you're thinking about your different parts, like inputs, think about, you know, what they would need to actually run this to do it. Activities, what are they actually doing? You know, what's the main thing they're doing as part of this program. Outputs, that's things like the number of clients served or the number of sessions or completion rates. And then outcomes, again, what are short term things you'd want to see, intermediate things you'd want to see, and long term things you'd want to see based on your outcome of choice. So even though people in this class are going to be choosing some of the same agencies. Obviously, we are only dealing with three agencies. There can be different outcomes for the agency. So, you know, that's where you'll do a little investigative work. Okay. Oh, I'm glad it's making more sense to you, Lo. Yeah. All right. Any other? Yes. For the figure you're referring to on page 28, is that the same that you were showing a while ago? Yes. Okay. Because, yes, I ordered the book. It didn't get to me yet. Oh, okay. Um, I did find, um, chapter one online. Um, well, I mean, I have the book, I have the ebook, um, obviously as a faculty member, I get the ebook, you know, the book or the actual book, but I did find chapter one online. So if you, if you don't have it yet and you need that let me know and i i think i would be able to share that with you okay thank you so much sure i got this chapter also on pdf for free but the chapter one and two the other chapters know yeah yeah so a lot of times publishers will share chapter one or chapter one and two um That's why I'm okay with doing that. Obviously, I don't want to do anything ******* or anything. But I think a lot of times publishers will share or make open access chapters one or chapters one and two, because, you know, even if students order the book, they might not get it right away. So I think that's not an issue. um all right um any other questions or comments not i guess i will just talk about the extra credit um so for um the um oh that's reasonable now that's good to hear that the ebook's 51. I know some of these books are like hundreds of dollars, so I'm glad to hear this one's not too crazy. 51 seems reasonable. But anyway, for the extra credit, I'm going to track your attendance each week. And at the end, in week eight, I will allow you to get some extra credit points. I think it's five points on one assignment, not assignment seven, because that's a bigger assignment. But for assignments one, weeks one through six, you will get to choose. Well, I guess you won't get to choose. I'll just choose the lowest score and I will add it to your lowest score. Okay, so it'll be a way of increasing. So you can get up to five points, but it will depend on, you know, the sessions you attend and whatnot. But it's a way to kind of help if you're kind of like a borderline grade in the class. And what else? So for the discussions, you're only required to do two responses. I did make a change to the class. Well, I'm making a number of changes to the class. But one change I made that I think you'll be happy about is that for week five, because it's Thanksgiving week, there's no assignment that week. so you'll start working on your analysis and your descriptive statistics and that'll be like part of your discussion and then of course you can work on more things with your analysis um which you will be doing for week six but i don't want to have to you guys to have to do um a formal assignment for week five when it's um thanksgiving um so i think that's the main thing and then yeah i will be releasing um right now i have the first two weeks open and then the next weeks will always be open at least the friday before that that week um but i will probably release like two weeks at a time but i just want to see how it's going um obviously if anybody has any feedback or any questions or concerns let me know um but i'm trying to simplify it and make it like very doable and easy to to go through the class but still learning and still growing really figuring out how to do a program evaluation so um all right any questions or comments before we wrap up not here okay um thank you no problem well thank you guys for coming um and uh as you go through just feel free to reach out if you have any questions um i'm happy to help out i know the logic model's not something you guys have ever done before. So just do your best. It's just a one -page document. Obviously, you can't overload those boxes. You want to make it, you know, just kind of clear and concise and organized of what the program's attempting to do. All right. Thanks. I hope you guys have a great week. I think this will be a fun class. Thank you. All right. Good night. Have a good night. Have a good night. Have a good night.