PP Presentation

Program Evaluation: Results and Outcomes

Program Evaluation Outcomes

Michelle Ellis, MA

The results of the CELA test are communicated actually through the newspaper and the news, that is kind of public

information as to how diverse the school community is, along with CSAT results also the language proficiency and

population of a school and their English Language Learners is publicly available, as well. And this is something that

I think we could probably work on because many parents do not speak English, and we mail the results home with

English, Vietnamese and Spanish letters to each family so that they will understand their student's abilities in these

four modalities that I spoke of.

And this testing begins in kindergarten and can go all the way through 12th grade. So conceivably a student might

be receiving ESL services for the entirety of their schooling career, but typically students and their parents will find

out that they are creeping along and getting better and better at these four modalities and get to see on a scale of

beginning, early intermediate, intermediate, early advanced and advanced as to how well they are doing with their

English proficiency.

This helps families to be able to guide their students and you need to be working, for example, on your reading

skills, because look at your writing score, or look at your listening and speaking score, those are really doing well,

but your reading needs some work. More often than not a student's writing score will be behind that of their other

scores, and teachers, parents and the students can have collegial conversation to try and figure out how to best meet

the needs so that a student's writing skills can improve and be that of a native speaker of English and a native writer

of English.

At a district level, all of the ESL teachers get together on a regular basis, we have a monthly meeting and throughout

the year we discuss the previous year's result and the next year's plan and evaluate between schools or feeder

schools. For example, if a middle school has particular results and the high school does not seem to be meeting the

needs or progressing at the same level, middle school and high school teachers will get together and talk about the

specifics that has been implemented for a particular student to help ensure that that transition between middle and

high school is smooth and that a student does not get lost in the shuffle.

Though I have some opinions about CELA testing and how much time it takes away from what we do

instructionally and creatively in our classrooms, I believe that overall this exam, along with the other exams and

assessments that are done really help us to help students be successful.

Nathan Long, EdD

The question really arose: Were we happy with the outcome of the total analysis? And initially, yes. I mean, if we

are looking at the program evaluation of the writing center from one point in time, it looked really good. However,

this formative Program Evaluation approach was critical, because in this first year we were able to see what was

working, what patterns we are developing, and it served as a critical juncture for us in identifying new outcomes for

the program. And herein lies for us the power of program evaluation at the collegiate level, especially in

understanding our own programs and own services, and what it means to the development of better indicators for

growth, program improvement and data collection.

So my role in facilitating or participating in the process was to ensure that the data was collected, that our staff and

myself met and reviewed that data, and analyzed it, and then came up with some actions for future work. Now as we

presented our results to the stakeholders in the process, including our faculty and students, they all seemed to nod

their head in agreement, "Yes. Absolutely. The writing center has great value here. We are glad it is here." But we

really think that these outcomes make better sense.

And those outcomes which we particularly crafted out of that data, focused specifically on two major components.

Better quantitative indicators needed to show satisfaction or high satisfaction with writing center staff support. We

also needed to show a stronger correlation between writing center activities and support and academic achievement.

And looking at what the faculty said—absolutely. Faculty want to know that the services provided are in fact

helping students, and if we look back historically at accreditation effort in colleges and universities, as well as K-12

and other educational entities, one of the things we are now moving towards is a better understanding of how our

students learn, and how we can prove that. What evidence do we have that our students are actually improving?

And so these new outcomes that we identified, we hope will better answer those questions, so that when we are up

for accreditation, when we are working on program improvement, we are able to demonstrate to our community,

stakeholders, our students and faculty, that this program has value.

Things that take a little longer than expected or anticipated, and one of the things that came out of this, is we felt we

could summatively evaluate, but we realized we were going to need to more formatively evaluate, and stretch out

the evaluation process over the next few years. So the data was not just collected in a one-time vacuum and

informed everything we did, but actually served as sort of a benchmark in evaluating the work that has being done

by the writing center.

So in the sense, does it make a difference in terms of how long things took? Absolutely. But did it make a positive

difference? Yes. Because now we are analyzing data more holistically, we are looking at the program long term as

opposed to short term, and we are using outcomes that we believe are more measurable and better monitored, and

evaluate the progress of our students and our writing center staff, in terms of the support they provide.

I do not know that I would have changed anything about the process related to Program Evaluation at this point. I

think one of the things that most program evaluators realize is that this process really is organic, especially in

program evaluation, you are really looking at how things emerge and evolve. Sometimes your best methodological

approach may change as you are going through analysis of the data. You may find that you need absolutely more

data in order to make better decisions about whether a program should stay or go, what type of improvements to

make, etcetera.

I might have changed the approach to establishing clear, more measurable outcomes at the onset, but I think again as

a professional, I also realized that my limitations are that I do not know everything. And that one of the things that

helped us in terms of the data we received in the program evaluation that we initially did, was that our outcomes

were developed out of the data. We were able to really pull those out and have better outcomes as a result of the

initial data collection.

Robert Wang, PhD

Examples of evaluations that I have been involved with: Probably the most recent was an evaluation on a field-based

environmental science program on the Chippewa/Cree Reservation in Northern Montana. This program basically

blended Native and Western Knowledge Systems related to science. And with regard to the Program Evaluation, we

are really looking at mixed methods, and we are specifically looking at student efficacy and creating an

ethnographic study of the program.

I was really lucky with that particular evaluation, I was serving as a Principal Investigator and I was able to higher

an excellent evaluation team that pretty much looped with their participants during the program, and had a very

strong relationship and rapport with the participants.

As far as stakeholders on this particular project, it was really the funding agency which was the National Science

Foundation, and also the tribe. The tribe members because obviously things don't go well, then it is more difficult to

get continuing projects like this on that reservation.

Again, my role in that particular project was as a Principal Investigator which really meant that buck had to stop

with me. So I had a overall supervision of the entire project including evaluation team. What that translates to is

making sure that evaluation team has all the tools that they need to do a good evaluation. And when you are working

for example on a reservation that includes logistics, but also making sure that they have access to for example, tribal

leaders and elders, and folks that they want to sit down and talk with. And making sure that they knew the norms as

far as how to interact with those individuals.

I guess my general take on that also is to work with the evaluation team, but make sure that I do not micro-manage

the evaluation team because they are to do an independent assessment. But I am always balancing that with the fact

that time is money, and I need them, being the evaluation team, to stay focused and organized, otherwise it is very

easy to blow a budget just on evaluation.

Alright, let us see, go with the outcome of this evaluation? basically the methodology—the specific methodology we

did, again a mix study where they are looking at quantitative data, basic demographic information, student test

scores, etcetera. And then they also did small interviews in focus groups to get that in the graphic information.

Was I happy with the outcome of this? Yes, I was, and I think a lot of that had to do with the experience of the team

doing the evaluation. I think the most meaningful information actually came out of the qualitative portion of the

study rather than the quantitative. In this particular case that is not always true, in that program evaluation.

How did I participate within this program? Again, I guess my general philosophy is to make sure that the evaluation

team has all the tools they need to do their work, but not to get too involved in it, or try to prejudice the results in

one way or the other. So once the evaluation team gets going, I really try to make sure that I am kind of removed

from it since I am also serving as a Principal Investigators.

On this particular project, the stakeholders were pretty positive as far as the results both folks from the tribe and the

funding agency were happy with the results. And just like many Program Evaluations, yes, it did take more time

which equates to more dollars having to be spent. But I think investing that money into evaluation is a real positive

thing in the sense that it ensures that project goals are met.

One thing that could have probably been done better with this particular project is wider dissemination of results.

That is always an issue, because that usually comes toward the tail-end of any type of problem evaluation.

Recommendations

Michelle Ellis, MA

The only improvement that I could see that would be really beneficial is receiving the results for both CELA test and

CSAP test in a more timely manner, prior to the very end of school or even in some cases over the summer, we are

incapable of actually getting together and making changes to the next year in a lot of cases. So that would be the

only improvement that I could see.

Overall it has been a real pleasure of mine to get to be a part of the English Language Department at Bear Creek

High School and over the years, starting out kind of accidentally in ESL as a science teacher, who was just given an

ESL earth science class, my first year of teaching has grown into a passion of mine to get to be involved with

English Language Learners and their families, and try and help develop programs and evaluate assessments that will

be effective in not just meeting a quota or giving a test that is required, but instead using those exams, those

assessments to effectively help tailor every program to individual student needs.

Nathan Long, EdD

In terms of general advice, I would say, I think in Program Evaluation one of the key components is being clear with

your stakeholders, and with the folks that you are working with to evaluate a particular program. And going back to

that initial discussion about outcomes, see if you can establish really strong measurable outcomes at the onset, that is

a key component in developing this out. Some other advice that I would provide—and one piece of advice that I

have been using myself is to have outside evaluators come in.

What we are now doing at our institution, is to bring folks from other institutions or universities to evaluate the data

that we have collected, and also to provide input. And you might call that, in Program Evaluation, an external

auditor. That external auditor is vital to providing an outside bird's eye view of your program and to say, hey, here is

where we think that data is pointing to, and you might want to consider these other elements. In fact, our writing

center director has enlisted the help of a major university writing center director. And she is bringing that person in

to review processes that we have in place currently, to see if we can better improve in meeting our new outcomes.

So with that said, Program Evaluation has a lot of intrinsic and extrinsic value in serving both the community, the

stakeholders that are involved in terms of your institution or organization, and can really benefit you in terms of the

long-term goals and objectives that you set forward for your institution.

Robert Wang

What advice can I give to folks doing evaluation? I think the key to evaluation is balance. We need to be able to fit

the overall goals of the project. I know a lot of folks involved in Project Evaluation that want huge percentages of

the budget, and that becomes cumbersome because no funding agencies every going to fund a project where the

majority of the money is spent toward Program Evaluation.

So I guess one of the things my advice would be for people doing evaluation to make sure that you have a balanced

perspective that you are sensitive to the overall project goals. But at the same time, you hold your ground and as far

as making sure that it is quality work. Some conclusions: Just like traditional qualitative research, many times you

have on emergent design with your research, as far as your research approach. But at the same time when you are

doing Program Evaluation, you also have to have a clear plan on what the ultimate goal of the evaluation is, yet need

to balance those two things.

And finally, last but not least, I think just like in business, communication is key, so if you are involved in Program

Evaluation of your program evaluator to be in contact and communication with the principal investigators and other

folks on the project team, I think it is pretty critical. So I guess that is it. I guess that is in a nutshell my experience

with Program Evaluation.

Credits

Subject Matter Expert:

Heather Miller

Interactive Design:

Marc Ashmore

Instructional Design:

Nicole Hettwer

Project Management:

Erin Coffey Licensed under a Creative Commons Attribution 3.0 License.