Case Study

[music playing] LAURA HADDOCK: Regardless of

the setting that any counselor works m there is an

ethical responsibility to periodically evaluate the

effectiveness of services. Activities that support

a counselor's role include program development,

program evaluation, and needs assessment. And counselors will conduct

planned, systematic evaluations of counseling programs

on an ongoing basis to determine the relevance and

effectiveness of the program. The evaluation should

include a description of the client competencies or

outcomes that were intended-- how you intend to collect

data and the rationale for a chosen data collection. You want to know whether

the therapist accomplished what they set out to accomplish

and whether the clients took away what the therapist

wanted them to take away. The information that you

gather will help the therapist in understanding the strengths

and weaknesses of the service delivery and make sure

that the counseling program goals are aligned to the

individual goals of the client.

· Walden University, LLC. (Producer). (2017).  Program evaluation [Video file]. Baltimore, MD: Author. Note: The approximate length of this media piece is 18 minutes.

Transcript

[music playing] RICHARD BALKIN: As a

practitioner, or even as a researcher,

you're going to be doing program evaluation,

because essentially, what you need to be able to do,

is to be able to identify the fact that what you're doing,

whatever services that you're providing, are

actually effective. In the mental health

world, you want to know that your clients

are getting better with the programs and

services that you're using. In terms of school

counseling, you want to know that your practice,

your job responsibilities, add to accountability and

success of the students you work with. So, in this respect,

program evaluation ties hand-in-hand to

evidence-based practice. What are the models that you

utilize that are known to work? Or can you show that the models

that you are using actually do work? The program I work with is

the Antonio Garcia Center. It's a program that services

court- referred youth. We provide family counseling

services, family intervention. We also provide counseling

services for the youth. And, as you can imagine,

with a variety of problems that core referred youth

have-- such as gang involvement, substance abuse,

difficulty with school, truancy, for example,

maybe family dysfunction has led to shoplifting or

other antisocial behaviors-- that with the

variety of programs we want to make sure that the

youth that we serve actually do get better. And how do we evaluate that? Well, let's take you through

a program evaluation model. This model is known as

the Accountability Bridge. And in this model,

we'll take you through the specific

steps that move into how we know whether or

not the services that we're delivering are

actually effective, and maybe even contribute

to evidence-based practice. What we have with the

Accountability Bridge Model is a cyclical model. You can start in multiple

places on the model, but for the purposes

of this exercise, we'll start with

program planning. And program planning,

we're simply wanting to think about what

are the services that we're going to be providing

for the youth in order to address a particular problem? For example, maybe

we have a high degree of family conflict with

the kids that we serve. And so we're going

to offer programs related to conflict resolution. Maybe we notice that

a lot of our kids have problems with

controlling their anger, so perhaps we have

some type of program that addresses anger management. And so what we do is we

plan for the specific area that we wish to address. What are the

psychoeducational components that go into teaching

anger management? What are the

psychoeducational components that go into teaching about

mediating family conflict, or even resolving it? So, for a program

planning perspective, we think about both the

psychoeducational components and the process

components that we want to include in the program. The next step is

program implementation. You need to consider

the fact that, OK, we have these youth

who were referred by the court for services. Now, how are we going

to do this program? When are we going to offer it? For example, is this

a weekend program, or does this happen

on weeknights? How will kids get

transported to the program? How will they be able to come? How long will the program last? How many weeks will we

provide these services? Who is going to

provide the services, and what training do they

have to provide the services? These are the aspects of

program implementation. And then, what

exactly are we doing? As I mentioned

before, if we're doing a psycheducational

component, actually providing the

necessary materials for the psychoeducational

component. Or, having qualified

practitioners to process with families

and youth about the problems that they may be

experiencing in their home or with their families. As we are providing

these programs, we want to monitor

exactly what's occurring. Do we see kids engaging, and

do we see families engaging? Are we noticing any changes? Does the information

that we provide seem particularly useful, and if

so, how are we assessing that? Maybe we are

monitoring the program by providing some

formative evaluations throughout the

process, evaluations that happen across the process. Maybe they're questionnaires,

maybe they're surveys, maybe they're instruments. However, when

compared to kids who also had a history of offenses

but did not receive services at the Garcia Center, we notice

those kids, over half of them had a history of reoffending. So, we were able to say

to our stakeholders, look, our program is effective. Kids who come

through our services are more likely not to reoffend. That's an important

summit of evaluation. Naturally, what

we want to do now is we want to be

able to bridge what occurs in our program

with the context, with who's going to be

utilizing the services, and how is this

information beneficial. So, as part of any

scholarly practice, it's not sufficient just

to have these findings. We also have to

get the word out. And in our particular case,

with the Garcia center, it was good news. Our program seems to

be working, and we want to let our

stakeholders know, hey, our program is working. So we arrange meetings with

the juvenile probation offices and perhaps with the

judges who are invested in using our program for

court- referred youth. And we let them know that we

seem to be making an impact, that kids who get

referred to our program, by and large, the majority

of them will not reoffend. So, once our stakeholders

are aware of the good work that we're doing, we also want

to get feedback from them. Perhaps they have other problems

they want us to address, or perhaps they

want to make sure that we're able to continue the

services that we're offering. So, it's important for us to

be able to either verbally or even in writing, or

perhaps through survey, be able to find

out what they feel about our program and the

services that we've provided. Naturally, with the

judges that we work with, they were pretty happy

that they were not seeing the same kids

over and over again, that the kids seem to be

genuinely doing better. What we have found

working with the juvenile courts is that there's no

single problem to resolve, but more many multifaceted

problems to address. Perhaps we need to

address graffiti, or perhaps we need to

address gang involvement, or perhaps we need to address

anti-social behavior, or school truancy. So, the judges give

us a lot of feedback in terms of what problems

they see in the process. At this point, they

begin to say, well, what services can you provide? Here's what I think

we're needing. In particular, one

of the judge said, we need something

for gang involvement. We're having more and more

youth involved with gangs. And the response from

the Garcia Center is, well, let's figure

out what we can do. So, with the feedback that

we get from the judges, it ties into, what are

the needs the community? So, we begin to both

strategize and identify what needs the community has. With respect to

gang involvement, we can see whether

or not there was a rise in anti-social

activity or violent crimes as it relates to gangs. We may need to address issues

related to substance abuse, or we may need to address issues

of weapons and gun control. There's a variety

of needs that can be addressed with

gang involvement. So, one question is,

how are the youth getting a hold of

firearms, weapons, that bring about

more violent crime? So, we begin to assess the

needs of the community, find out how we can address

some of these problems, and hopefully

figure out how we'll be able to get

youth who would be appropriate for these

services to participate. As you can imagine,

kids who have highly anti-social

behaviors may be difficult in terms

of getting them into a counseling

group or session. So, just seeing not only the

plan, the needs the community and the needs the

youth, so that they have some buy-in into the

services that we provide is essential. For us at the Garcia

Center, we had to be able to identify

what was realistic. One of the things that we did

is we looked at model programs to identify, well, what

are successful programs who deal with gang involvement? What are they doing? And we figured out

from those programs which of those service

objectives meet our program. In other words, can our program

meet the following service objectives and make a difference

with gang involvement? So, that's where we're at today. We're identifying

additional objectives to develop programs

to work with kids who have affiliated with gangs. And, in that process,

it'll bring us back into the program

evaluation cycle, in terms of eventually

designing a program to address the needs of the

youth and implementing it. And you can see the program

starts all over again. The cool thing about

program evaluation is that's a great way

to inform practice. In other words, it tells us

whether or not what we're doing is working. With program

evaluation, it really ties nicely into

evidence-based practice. Keep in mind, while

not trying to bias one toward quantitative

research, in most cases when we talk about

evidence-based practice, we're talking about

quantitative findings that justify such practice. And the reason is this. Many programs that

go under evaluation may, for example,

be grant funded, or the stakeholders are looking

for clear data that says, this works. And from a generalizability

prospective, quantitative research fits that model

almost more strongly than qualitative research. That does not discount the

need for qualitative data in program evaluation,

but you may be challenged with

how to objectify that, so you can say that these

findings work effectively across the board and a

variety of populations. So, in program evaluation,

the implications are twofold. One, for the program

and the stakeholders, they obtain data that

identifies whether or not their product is working. From a more global perspective,

hopefully the program evaluation can be made

into a model program and published, so that

other programs can examine, hey, here's something

successful that's going on. Let's see if we

can implement this. To kind of give you

an anecdotal story, I was able to develop

an instrument that was published in a counseling

journal back in 2007. And what was neat about that

publication is being contacted by psychiatric hospital

in Wisconsin that said, hey we saw your article. We saw your instrument. Could we use that

at our hospital? And it provided an

opportunity for us to share data and talk

about how therapeutic goal attainment might

work for adolescents, not just in the psychiatric

hospitals I studied, but across the country. So, it's a neat opportunity

when you can get a program out there and have an

evaluation model that can be generalizable to

other people and programs. Let's talk about some

components to look at when you find a

program evaluation article and want to know how

to implement this into my practice. Naturally, the first

thing to look at is is, how does this

program evaluation piece tie into what you do? In any particular program, there

are probably aspects that will fit your program, and then

maybe aspects that don't. So, try and identify

what those are. Is it the entire

program that you want to bring into

your practice, or is it just elements? Also look at how those

elements, then, were isolated. In terms of evaluation, how

were the programmatic pieces evaluated both quantitatively

or qualitatively? Were the pieces placed

together as an entire model? In that case, you really

can't separate them. Or were they identified

by each element? In which case you can

piecemeal it together. Also, remember to

always evaluate the nature of the data. Did they identify

statistical significance as a result of

using this program, and were the

relationships meaningful? In other words, did you evaluate

both statistical and practical significance of

what was evaluated? So, remember that statistical

significance doesn't make it meaningful,

and you always have to keep those

two aspects separate. In terms of qualitative data,

is what was used transferable? Was it coherent,

insightful, and useful? When looking at a program

evaluation qualitatively, keep in mind, is it coherent? Is it insightful? Is it useful? And therefore maybe it's

transferable to what you're doing. This ties into the ethical,

legal, and multicultural implications to

program evaluation. Always keep in mind,

is the sample-- the participants who engaged

in this particular program-- a match for the participants

that you deal with? For example, in South

Texas at the Garcia Center, most of the youth

referred to our services are Latino or Latina youth. So, this is an

important consideration when we're providing services. We want to make sure that

the family system that we're dealing with is

appropriate, and that we're hitting on the cultural issues

that mainly Latino and Latina families deal with. Additionally, you want to

consider whether or not these programs are a good fit

for the population you serve. As with any program evaluation,

the stakeholders probably have a vested interest in making

an impact on the community. So, ask yourself,

who's in my community? Does this program serve

everybody in my community, or does it provide services

only for a particular subset? Another example

is that we started providing services specifically

geared toward at-risk youth who are female. Why? Because the courts

identified a need to begin to address

antisocial behavior among female at-risk youth. There we've seen a rise

with this particular group. So, we developed

specific programs related to adolescent females

in anti-social behavior. Keep in mind that

program evaluation can encompass a series

of research designs, methodologies, and also

employer accounts and skills. It truly puts you

in the driver's seat of being able to deliver

services, identify community needs, talk with stakeholders,

and use multiple methodologies in identifying whether what

you're doing is working and is it making an impact. In this respect, perhaps there's

not enough program evaluation research being published in

the counseling literature. There isn't a lot of

evidence-based practice that we see, and

one of the reasons is, is because the

primary people who should be doing evidence-based

practice research aren't the scholars

in the field. They're the practitioners who

are right there on ground zero delivering the services. They're the ones who

are most equipped to be doing program

evaluation and identifying evidence-based practice. So, hopefully you'll see

this as maybe a call to arms, so to say, that

as a practitioner, it's important to not just

to be delivering services, but to also be

thinking about, how can I demonstrate that

what I'm doing is working?

LAURA HADDOCK: Regardless of the setting that any counselor

00:10