discussion4-2

Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=utne20

The New Educator

ISSN: 1547-688X (Print) 1549-9243 (Online) Journal homepage: https://www.tandfonline.com/loi/utne20

Using Internal and External Evaluation to Shape Teacher Preparation Curriculum: A Model for Continuous Program Improvement

Patricia J. Norman & Sara A. S. Sherwood

To cite this article: Patricia J. Norman & Sara A. S. Sherwood (2015) Using Internal and External Evaluation to Shape Teacher Preparation Curriculum: A Model for Continuous Program Improvement, The New Educator, 11:1, 4-23, DOI: 10.1080/1547688X.2015.1001263

To link to this article: https://doi.org/10.1080/1547688X.2015.1001263

Published online: 12 Feb 2015.

Submit your article to this journal

Article views: 533

View related articles

View Crossmark data

Citing articles: 2 View citing articles

The New Educator, 11:4–23, 2015 Copyright © CCNY and ATE ISSN: 1547-688X print/1549-9243 online DOI: 10.1080/1547688X.2015.1001263

Using Internal and External Evaluation to Shape Teacher Preparation Curriculum: A

Model for Continuous Program Improvement

PATRICIA J. NORMAN AND SARA A. S. SHERWOOD Trinity University, San Antonio, Texas, USA

The same high stakes accountability measures that have shaped the K-12 educational context over the past decade have become part of the university teacher preparation landscape. National accrediting organizations have created more rigorous, out- comes-based accountability measures for teacher preparation pro- grams. Educational stakeholders offer mixed reviews about the value and impact of these accreditation efforts. In this article, however, we argue that such external evaluation measures when paired with our own internal evaluation processes have gener- ated meaningful changes to our elementary teacher preparation program.

INTRODUCTION

Although well-crafted teacher preparation programs impact preservice teacher learning (Brouwer & Korthagen, 2005; Hart, 2002; Levin, 2003; Yao, 2010), ongoing questions about the effectiveness of teacher preparation pre- vail (Greenberg & Walsh, 2010; Washburn, Joshi, & Cantrell, 2011). The same high stakes accountability measures (NCLB, 2008) that have shaped the k-12 educational context over the past decade have become part of the university teacher preparation landscape. Concerned about the perceived inadequacy of teacher preparation, the U.S. Secretary of Education has urged programs “to make better outcomes for students the overarching mission that propels all their efforts” (Duncan, 2009).

In response to these concerns, organizations like the National Council for the Accreditation of Teacher Education (NCATE) and the Teacher

Address correspondence to Patricia J. Norman, Trinity University, One Trinity Place, San Antonio, TX 78212-7200, USA. E-mail: [email protected]

4

Using Internal and External Evaluation 5

Education Accreditation Council (TEAC)— now consolidated to form the Council for Accreditation of Educator Preparation (CAEP)— have created more rigorous, outcomes-based accountability measures for teacher prepa- ration programs (NCATE, 2008). These standards assess programs across multiple domains and require them to develop and implement assess- ments systems that clearly demonstrate that their candidates have developed the knowledge, skills and dispositions to be effective classroom teachers. In addition, these organizations routinely review and revise their standards to reflect the latest research in best practices in teacher preparation and teaching (Curtis, Bordelon & Teitelbaum, 2010).

Educational stakeholders offer mixed reviews about the value and impact of NCATE (Dillon & Silva, 2011).1 Some find NCATE accreditation a meaningful process that encourages programs to set priorities and to monitor progress (Dhonau, 2007; Meyer, 2010); others describe NCATE’s standards as internally inconsistent and ineffective in improving teacher train- ing and quality (Fein, 2004; Levine, 2006). We find that NCATE’s requirements and standards have served to strengthen our already nationally recognized program.2 Notably, the assessment process helps to systematize the data collection process and improvement strategies already in place within our department.

This article describes and analyzes how combining NCATE’s external evaluation with our own internal evaluation processes has generated mean- ingful changes to the elementary teacher preparation program. In particular, we describe our approach to continuous program improvement, offering examples of how both internal and external evaluation measures have helped us strengthen the elementary program. We conclude by exploring the benefits and challenges encountered through the process.

TRINITY UNIVERSITY’S MASTER OF ARTS IN TEACHING PROGRAM

Trinity University is a small, private liberal arts university in the south central United States with an enrollment of 2,600 students. Trinity serves a predom- inately White (62%) student body whose median SAT score nears 1300. The Education Department at Trinity offers a five-year Master of Arts in Teaching (MAT) program leading to certification in one of three levels: (a) early child- hood through grade six, (b) grades four through eight, and (c) grades eight through twelve. As undergraduates, students at Trinity major in a content

1Given the recent consolidation of NCATE and TEAC into CAEP in July 2013, the impact of this new accreditation agency is not yet known.

2Although NCATE is now part of CAEP, our teacher preparation program is accredited by NCATE through spring, 2018.

6 P. J. Norman and S. A. S. Sherwood

area and take a dozen hours of supporting education coursework including two practica, one field seminar, one human growth and development course, and one special education course. Thus there is no undergraduate major in education offered. After receiving their bachelor’s degree, students enroll in the Department’s one-year MAT program where they complete a five-week summer program before entering a one-year, unpaid, mentored internship at a local elementary school. Throughout their internship, candidates com- plete graduate coursework designed to introduce them to best practices as well as help them both prepare for and make sense of their field-based experiences.

The Education Department typically graduates 35-45 students per year across all three teacher preparation programs and 10-15 within the elemen- tary program. Historically, the program has had a 100 percent pass rate on its state certification exams and 100 percent placement of its students in teaching jobs after graduation. Given the small, “boutique” nature of Trinity’s three preparation programs, we find it relatively easy to make programmatic changes since there are so few faculty who teach within the MAT program. Pat has served as the elementary program coordinator for the past thirteen years, teaching two required undergraduate courses, four of six required graduate courses, and supervising students in their year-long internships. Sara has taught as an adjunct professor in the elementary program for eight years as both an undergraduate and graduate instructor and university field supervisor to MAT students.

CONTINUOUS PROGRAM IMPROVEMENT MODEL

As illustrated in Figure 1, internal and external evaluations run parallel to one another, and both impact decisions about how to maintain, refine and revise the program. Our model for continuous program improvement begins with the EC-6 MAT curriculum that is implemented across the graduate students’ ten-month program. The curriculum generates a pool of yearly collected data that drives both internal and external evaluation. In the case of inter- nal evaluation, we routinely collect a variety of data to support ongoing assessment and program development including assessment results of candi- date work, student/instructor course evaluations, written reflections gathered at the end of every weekly graduate seminar, and an end-of-year focus group interview held with the elementary MAT candidates. Selected data, including information gathered from our own direct experience teaching the curriculum, is then used to drive our analysis and reflection. For instance, the two of us might engage in a conversation to discuss the feedback gen- erated during the candidates’ focus group interview. Those conversations and independent analyses lead us to explore implications for programmatic

Using Internal and External Evaluation 7

In t e

r n

a l

E x

t e

r n

a l

Exploration of Implications

Synthesis of Data

External Feedback

Data Analysis and Reflection

Maintenance, Refinement or Revision of Curriculum

Data Pool

Implementation of Early Childhood-Grade 6

Master of Arts in Teaching (MAT) Curriculum

FIGURE 1. Model for Continuous Program Improvement.

change. At this stage, possibilities for revision are identified and a deci- sion is ultimately made to implement an explicit revision, refine an existing curricular component, or maintain an aspect of the program as is.

In the case of external evaluation, NCATE requires that programs address individual specialty areas, such as the elementary standards designed by the Association for Childhood Education International (ACEI). To address these standards, we collect data from a minimum of six different assessments that address the following: licensure, content knowledge in elementary edu- cation, candidates’ ability to plan instruction, student teaching, candidates’ effect on student learning, and an additional assessment that addresses ACEI’s standards. The specific data that we collect to address these six assessment areas include: the state’s Generalist Early Childhood – Grade 6 teacher certification exam; grade point average on undergraduate content coursework or content exam such as the Praxis; a curricular unit that uses Understanding by Design principles; an exit portfolio that addresses Trinity’s Standards for Professional Practice; a Teacher Work Sample; a Dispositional Assessment; and a set of Teaching for Understanding entries that demon- strates candidates’ ability to plan, teach and assess students’ learning in all content areas.

8 P. J. Norman and S. A. S. Sherwood

Data analysis and synthesis are largely driven by NCATE requirements and include:

● A brief description of the assessment and its use in the program; ● A description of how each assessment specifically aligns with the

standards it cites; ● A brief analysis of the data findings; ● An interpretation of how that data provides evidence for meeting

standards; and ● Assessment documentation, including assessment tool itself; the scoring

guide for the assessment; and charts that provide candidate data derived from the assessment (NCATE, 2013).

ACEI formally reviews the program’s formal synthesis of data and provides external feedback, which we explore and use to maintain, refine or revise our program.

In the following two sections, we provide specific examples of how internal and external evaluation drive program improvement in Trinity’s elementary MAT program.

INTERNAL EVALUATION: ANOTHER LOOK AT CLASSROOM MANAGEMENT

A series of recent examples of how we have engaged in internal evaluation to strengthen the MAT elementary curriculum focuses on classroom man- agement. We believe that the best place to learn how to teach— including managing students’ behavior and learning— is in a classroom under the thoughtful guidance of a mentor teacher who can induct novices into the practice of teaching. Candidates are able to learn about management not only through their careful observation and study of their mentor’s practice but also through their ongoing efforts to establish their moral authority and manage students’ behavior and learning as they assume increased teaching responsi- bility across their year-long internship. Interns receive direct support in learn- ing to teach across the year from their mentor teacher and university supervi- sor. In addition, their graduate coursework is designed to help them prepare for and make sense of their field-based experiences. Thus one venue for sup- porting their ability to manage a classroom learning community is through their graduate coursework where we deliberately create space for candidates to examine best practices and reflect on their own emerging practice.

Ongoing activities and assessments connected to candidates’ graduate courses and their classroom-based internship create a diverse data pool to support internal evaluation. In the following sections, we define the spe- cific data we analyzed and how that reflection on the data led us to revise,

Using Internal and External Evaluation 9

refine and maintain different aspects of the MAT curriculum as it relates to classroom management.

An Example of Program Maintenance

An ongoing way that students reflect on and evaluate their classroom man- agement skills is through the weekly process of Checking In. In both the fall and spring semesters, the 10-15 interns in the elementary cohort attend a weekly three-hour seminar. One weekly ritual that we devote 20-30 minutes to is “Checking In.” This is a time for each candidate to share something for our collective examination and problem solving, or to get a different perspective. Interns are given a minute or two to collect their thoughts about one pressing issue, celebration, question, or concern connected to their classroom-based experience. Each intern first takes several minutes to describe a field-based experience then the group responds. They often describe difficult encounters with a particular child or the entire class that involve management issues. Their fellow interns and course instructor serve as helpful resources not only for making sense of what transpired but also possible next steps to take.

Several weeks into the fall semester, for example, an intern in a first grade classroom used Checking In to share a concern about a child in her classroom who was “in constant motion” and made continuous sounds throughout the day. The intern explained that sensory information seemed difficult for the child to manage. Although the child was a phenomenal reader, oral communication was challenging for her. In addition, transitions— be they between lessons or between different locations in the school— presented significant difficulty for her. The intern was concerned that several students were becoming increasingly frustrated with the child who some- times inadvertently interrupted their learning with the noises and body movements she made. In response to her concern, we suggested that learn- ing how to manage her own emotional response to a situation is an important part of helping other students manage their own responses. In addition, we gave her the titles of two picture books by Jennifer Veenendall, one about a boy with Autism and the other about a girl with a sensory processing disor- der. We encouraged her to read one or both books aloud to help the child’s classmates better understand children who may act different from them.

During Checking In the following month, the intern reported that she felt increasingly comfortable working with the student who had initially intimidated her. The intern now recognized what a wonderful sense of humor the child had, something she hadn’t initially appreciated or noticed. Although the intern had not shared the suggested picture books as read alouds, she had borrowed language from the picture books when privately consulting with two students who frequently grew frustrated with the child

10 P. J. Norman and S. A. S. Sherwood

making noises in order to understand how they could all learn and work together even though they may have different needs.

We have received consistently positive feedback about Checking In. For example, during a recent focus group interview held at the end of the MAT experience, one student stated: “I liked taking time to check in—it was a really important part of class. It gave everyone a chance to offer suggestions and get it all out there.” A second student chimed in:

Checking-in helped me to clarify my own thinking about what was going on in my classroom. Talking through a situation helped me better under- stand what was going on myself, and it also allowed others to support me with encouragement and suggestions. It was a regular opportunity to share, so you didn’t have to go asking for feedback or begging others for advice. You knew the time would be there every week, so it was a relief to not have to try to find time to have those important conversations. I always felt better after checking in because I felt like everyone was not only root- ing for me, but they were invested in my situation, and cared about the outcome and how I got there.

Because of interns’ steady appreciation for “Checking In” as well as our own observation that they frequently act on the feedback they receive once back in their classrooms, we have maintained this component. Checking In enables candidates to get support and feedback from both their peers and professors in order to grow their practice.

An Example of Program Refinement

In the fall semester, a second way that interns explore classroom manage- ment is through their first formal inquiry project called The First Days of School. Anchored by the question “How does my mentor teacher begin the process of creating a classroom learning community?” the inquiry addresses these sub-questions: (1) What kind of classroom learning community does my teacher value? Why are these qualities or features important? (2) How do the arrangement of space and the uses of time support this kind of learning community? (3) What rules, routines and procedures need to be established for the classroom to function smoothly? How does my teacher go about this during the first days/weeks of school? and (4) What kind of learning does my teacher value? How is this communicated to students?

Interns collect a variety of data during their investigation, including for- mally interviewing their mentor and observing their mentor and students. They create a map of the physical lay out of the classroom then use a proto- col to examine whether and how the decisions that a teacher makes about the physical environment reflect her values and beliefs about teaching and learning. In addition, interns study the uses of time in classrooms, comparing

Using Internal and External Evaluation 11

and contrasting the typical daily schedule in their mentors’ classrooms and how the mentors’ decisions about the uses of time reflect their priorities and values. Interns record and analyze how rules, routines and procedures are introduced and contribute to the functioning of the mentor’s classroom learning community. Finally, the interns investigate how a culture of learning is created in their mentor’s classroom, including whether and how students are encouraged to learn with and from each other. The inquiry culminates in a written reflective memo that interns formally submit for evaluation.

We have used the First Days of School inquiry project with elementary interns since 2001 and have analyzed data collected from a variety of sources including focus group interviews and course evaluations. For example, when asked in the formal course evaluation to “comment on the assignment from which you learned the most and why,” five of the eleven interns in a recent cohort named the First Days of School inquiry. One student wrote, “The First Days of School assignment helped me realize how much/little class community there was in my class and gave me an idea of what I want my ideal classroom to look like.” Another commented, “After drawing a map of my classroom and having to show it to other members of the class I realized how important set-up can be.” Yet another intern wrote, “I learned a great deal during the First Days of School Inquiry. I really had to examine my mentor’s actions. I learned that everything she says and does has a specific purpose.” Finally, one intern explained, “It was so rewarding to be able to synthesize the work we had done at the beginning of the year in our reflective memo, to recognize how our actions, teaching, and practice were affecting our students and the classroom community, and to discuss and think about what we could do next.”

While this formal data suggests that interns find this initial inquiry into classroom learning community a worthwhile learning opportunity, we have made refinements to this program element based on our own professional judgment. For example, we have cut down on the amount of formal writing required. In years past, we asked interns to formally write and submit com- mentary on the physical map of their mentor’s classroom, the way students’ time is scheduled in their classroom, and the rules/routines introduced in their classroom as well as whether/how a culture of learning is established. Because we have added additional teaching responsibilities and graduate coursework connected to the teaching of specific content in the fall semester, something had to give. Past experience has taught us that we need to take care in helping interns balance their internship-based responsibilities and their graduate course commitments. Thus we now use graduate class ses- sions to examine classroom maps, analyze the introduction of rules/routines, and together consider a set of questions designed to help interns explore the culture of learning in their mentor’s classroom. We no longer, however, require that interns submit written essays on these aspects. Instead, we only require them to submit a final reflective memo that includes both an analytic

12 P. J. Norman and S. A. S. Sherwood

description of what interns have come to understand about the emerging classroom community and 2-3 insights and/or questions that probe beneath the surface of classroom life.

An Example of Program Revision

Across their MAT experience, Trinity students engage in additional learning opportunities around classroom management. Until 2010, for example, we introduced interns to non-traditional approaches to classroom management during the five-week summer term that they complete before entering their internship. Specifically, interns studied Beyond Discipline: From Compliance to Community (Kohn, 2006) to launch our investigation. We did so out of the belief that most candidates would be steeped in traditional notions of management once they entered their internships. We wanted to offer an alter- native vision of doing “with” rather than “to” students before being immersed in punishment and reward systems both in their internships and their initial years of teaching.

An exit focus group interview with the 2010-11 cohort created an impor- tant data source to analyze. During the interview, candidates are asked to reflect on strengths of the MAT program and to offer suggestions for improve- ment. Candidates are given a few minutes to gather their thoughts before they share their ideas with one another while we script their responses. In addition, we often share our own ideas for program improvement at the end of the focus group interview to elicit candidates’ constructive feedback.

When asked what suggestions the candidates had for improving the program, a candidate noted, “Kohn was too radical.” Her classmate responded:

I liked Kohn because it was so radical but maybe balancing it with some other types would be helpful. Reading that right before going into the classroom held me back from doing some things I would have done management-wise. It was always in my mind, ‘Am I doing it the Alfie Kohn way?’ It would help to have more discussions on different philoso- phies of classroom management, and it might help to talk about our classroom management beliefs before reading Kohn.

Based in part on candidate feedback and subsequent conversations, we realized that introducing candidates to Kohn’s ideas before starting their internship created significant challenges for the MAT students. The stu- dents’ feedback enabled us to see that by starting with Kohn’s treatment of conventional management practices, the interns often found themselves placed in classrooms where traditional approaches to management were the norm. This left interns feeling uncomfortable adopting management practices based on punishments or rewards. However, they were unable to establish alternative practices, whether from fear of undermining their

Using Internal and External Evaluation 13

mentor teacher’s established rules and expectations and/or the lack of skills needed to do so.

Jointly reflecting on this feedback, we significantly revised the way we addressed classroom management across the program the following year. In the summer course, we now use Teaching Children to Care: Classroom Management for Ethical and Academic Growth K-8, by Ruth Charney (2002). Our intention is to equip interns with concrete manage- ment strategies that maintain children’s dignity. Our goal is to introduce candidates to a set of management practices that enable them to establish their moral authority, build classroom learning community, and establish classroom rules and student expectations. Once candidates begin their year- long internship, we support students’ exploration of classroom management in several ways. First, they continue to complete the First Days of School inquiry project. Second, candidates continue to share their own challenges in managing students’ behavior and learning during “Checking in” in the weekly graduate seminar. In addition, we assigned and discussed Learning to Trust: Transforming Difficult Elementary Classrooms Through Developmental Discipline (Watson, 2003) in the graduate seminar. The text chronicles how a classroom teacher struggles to enact principled management practices in her classroom. Finally, in the spring semester, after interns have had a chance to establish their moral authority and help their mentor maintain rules and pro- cedures for the smooth operation of the classroom, we examine Alfie Kohn’s less conventional ideas.

This revised curricular sequence seems to have worked quite well these past two years. From candidates’ written course feedback, they felt well pre- pared to begin their internship after the summer session. For example, one wrote: “I loved our examination of classroom management. It really helped me prepare for the internship. I now feel better with my authority in the class and the organization of the room.” Another commented: “I feel like we have talked about a variety of different management options and I am prepared and excited to take them into my classroom this fall.” In our own observations of interns’ classroom-based work during the fall semester, the interns seemed much more comfortable and able to establish their teacher presence and address discipline problems than interns have in years past. Moreover, during a recent exit focus group interview, one candidate men- tioned, “We took a really good look at classroom management and that was really helpful.”

EXTERNAL EVALUATION: PLANNING, TEACHING, AND ASSESSING IN THE CONTENT AREAS

In addition to continually collecting and reflecting on data to revise and improve the program internally, we have found that the NCATE accreditation

14 P. J. Norman and S. A. S. Sherwood

process creates another important opportunity, be it more formal and system- atic, to collect and reflect on data in order to strengthen the MAT program. Similar to our internal evaluation process, external evaluation begins with the gathering and analysis of data that grows from the curriculum. The dif- ference is that while we can select any data to analyze during the internal process, including our own professional judgment that grows out of our direct experience teaching the curriculum, ACEI/NCATE provide clear guide- lines about what data is to be collected and synthesized for formal reporting. Furthermore, ACEI/NCATE provides formal external feedback that we are required to address. In other words, during internal evaluation, when we get to the point of exploring implications for our program, this process is completely voluntarily whereas with external evaluation it is mandatory.

As stated earlier, NCATE requires programs to collect data from a mini- mum of six different assessments that address licensure, content knowledge in elementary education, candidates’ ability to plan instruction, student teach- ing, candidates’ effect on student learning, and an additional assessment that addresses the specific standards with which we align our program, in this case ACEI. We submitted Trinity’s elementary education program report to ACEI in March, 2010.

We received ACEI’s response, much of which was positive, four months later. In their national recognition report, they wrote:

The program is very well designed and has structured a thoughtful set of major assessment points. Results are strong, and program review of assessment data is systematic. Of particular note is the clarity with which assignments, assessments and the program are described to candidates (and program reviewers).

Against this glowing feedback, however, the reviewers identified a significant concern within the ACEI Curriculum standards. These standards assess can- didates in terms of their content knowledge of and ability to plan and teach reading, writing and oral language; science; mathematics; social studies; the arts; health education; and physical education. Specifically, the reviewers commented:

The program assessments show clearly the alignment between the stan- dard and the candidates’ assignments. The resulting data, however, are reported by the domain or section, where multiple ACEI standards are aligned to one domain or section. Activities or requirements within assess- ments must be set off in such a way that each task or requirement is specif- ically designed to measure competency on a single standard or standard element. Because the assessment and/or rubric are designed to evaluate performance across ACEI multiple standards in some cases, the resultant data cannot be used as evidence of successful candidate performance on individual standards or elements . . . Assessments in their entirety must

Using Internal and External Evaluation 15

demonstrate evidence that candidates “know, understand and use” the concepts in Standards 2.1-2.7. None of the assessments cited for this stan- dard evaluate candidates on curriculum-specific performance in terms of instructional planning or implementation.

In other words, ACEI reviewers raised concerns that although the assess- ments that we used— namely the state’s EC-6 Generalist and EC-6 Pedagogy and Professional Responsibilities certification exams—addressed all seven curricular areas, the assessments did not assess candidates’ ability to plan and teach those seven content areas.

Initially this feedback was challenging to digest. The state’s EC- 6 Generalist exam is designed to assess not only content knowledge across all curricular domains but also knowledge of instruction and assessment within those domains. Trinity elementary candidates have scored at least 23 points higher than the state average on this exam for the past three years. Moreover, our teacher candidates’ initial pass rate on both certification exams has been 100 percent for the past six years, much higher than the 80 percent initial pass rate that NCATE requires. How were we going to create seven separate assessments to evaluate candidates’ content knowledge of and abil- ity to plan and teach within ACEI’s identified seven content areas? This is exactly what we had to do if we wanted to move the elementary program from being “nationally recognized with conditions” through August, 2012 to being “nationally recognized” until 2017.

An Example of Program Improvement in Response to External Evaluation

The NCATE process gives programs up to eighteen months to address “con- ditions” identified in the national recognition report. After taking the first two months to process this feedback and develop a plan, we then imple- mented and refined it over the following fourteen months. We first identified ways that we already internally assess candidates’ ability to plan for, teach and assess student learning across specific content areas. For example, at the end of their internship, interns are required to develop an “exit port- folio” which includes a series of entries demonstrating that interns have met Trinity’s Standards for Professional Practice. These standards include “Planning for Students’ Learning” and “Teaching and Assessing: Engaging All Students in Learning.” For the previous six years, as part of this sum- mative assessment, interns had been required to develop four Teaching for Understanding entries in reading, mathematics, social studies and science. For each of the four entries, interns included a unit or lesson plan that they had developed and taught then described why this content was important for students to learn. In addition, they had to describe one or more of the learn- ing activities implemented, explaining how these tasks fit identified goals

16 P. J. Norman and S. A. S. Sherwood

and how learning differences were accommodated. Finally, candidates gath- ered several student work samples created from the lesson/unit, outlined criteria used to assess students’ performance, assessed each student, and then explained how this assessment of students’ work helped them further support children’s learning.

Because these four Teaching for Understanding entries were not inde- pendently assessed but were rather part of a summative assessment including additional exit portfolio entries, ACEI did not accept the exit portfolio as demonstrating interns’ ability to plan for, teach and assess students’ learn- ing in the seven content areas (reading, mathematics, social studies, science, health, physical education, and the arts). Thus we first decided to decouple the Teaching for Understanding entries from the exit portfolio.

In the first year, we spaced the four Teaching for Understanding entries across the spring semester as part of students’ field-based experience and graduate coursework. We revised the entry descriptions so that candidates had clearer expectations for what was required of these now stand-alone projects. In addition, we added a fifth entry—Integration—where interns are required to develop a lesson or unit that integrates health, physical education and the arts then to collect and assess students’ work generated during the lesson(s). Finally, we developed a single formal assessment rubric for all five entries with the following criteria: goals for student learning; discipline-specific unit or lesson plan; discipline-specific teaching methods; and assessment of student learning. Language was provided for what it means to approach, meet and exceed expectations across the four criteria.

Based on feedback we received from candidates during an exit focus group interview, we made several revisions the following year. First, we spread the five entries across the summer, fall and spring semesters after learning from the interns that it was too challenging to complete all five in a single semester on top of their classroom teaching responsibilities and completion of other graduate coursework. To do so, interns completed the reading Teaching for Understanding entry during the summer session which focuses on reading development, instruction and assessment. In addition, we revised the structure of one of their fall graduate courses so that each month was devoted to the examination of a particular content area: mathematics in September, social studies in October, and science in November. Candidates completed those three entries during that corresponding month in the fall. We also structured their classroom teaching responsibilities to coincide with the planning, teaching and assessment of those content areas by month as well. Candidates completed the Integration entry in the spring semester in order to leave the bulk of their attention to “lead teaching” where they assume primary responsibility for planning, teaching and assessing students’ learning for six consecutive weeks.

A final revision that we made in the second year was to stop approach- ing the Integration entry as a requirement that had no merit other than

Using Internal and External Evaluation 17

to meet ACEI/NCATE requirements. We have always prided ourselves on ensuring that program requirements are relevant, worthwhile and directly supportive of novices’ learning to teach. Initially, we could not see the bene- fit of asking interns to complete the Integration entry other than for external accreditation. However, the high quality lessons that they developed and taught in response to this new requirement completely changed our minds. As a brief example, an intern in a fourth grade classroom not only inte- grated health, the arts and PE but did so in the context of teaching reading. Her reading goal centered on helping students understand how to glean and use information in procedural texts and documents, including deter- mining the sequence of activities needed to carry out a procedure. After providing instruction on how to read a “how to” text, the intern introduced two strategies to manage stress (e.g. breath technique and personal mantra), thus incorporating health and PE goals. The text she provided them focused on how to create a piece of origami. Students were expected first to read through the text in its entirety before creating their own piece of origami by following the instructions. If they experienced frustration while creating the art, they were encouraged to take responsibility for their personal health and well being by using the breath and/or mantra technique the intern had taught them.

The intern drew on several forms of assessment to determine whether students had met her learning goals, including recording anecdotal notes. For example, the first step was for students to read the text from beginning to end before starting to create their own origami. The intern could observe right away who did or did not read the text before attempting their origami. She also assessed students’ reading comprehension by viewing the final origami product. In addition, students completed a self-assessment rubric. In terms of the health and PE goals, the intern observed whether the students used the stress reducing strategies before asking for her help. We observed the intern teach this lesson and along with the intern were struck by the high level of engagement that students demonstrated. The lesson provided a very practical and engaging way to not only explore and unpack procedural text but also to learn important PE and health content/skills.

After submitting the five Teaching for Understanding entries along with an analysis of data findings from the first two years of implementation to ACEI, last fall we received word that the EC-6 MAT program has become nationally recognized. In their response to our revisions, ACEI commented:

The report provided the necessary evidence to show that the area of candidates’ ability to understand and apply pedagogical and profes- sional content knowledge, skills, and dispositions had been carefully addressed and documented. This report shows that Trinity University used the information provided in the earlier ACEI SPA report to make signif- icant changes to improve their program and to respond to suggestions

18 P. J. Norman and S. A. S. Sherwood

for improvement. The changes have been implemented and the program meets the requirements . . .

DISCUSSION

Educators need both formative and summative feedback to improve the quality of their instruction and students’ learning (Marzano, 2010). Teacher educators are no different. As our model (see figure 1) demonstrates, although we draw from a diverse pool of data, the internal and external evaluation processes work together to inform our practice and strengthen the MAT program over time. Our own internal evaluation measures create important formative assessment points. In contrast, our external evaluation measures, namely the NCATE accreditation process, provide essential sum- mative feedback by asking us to track and analyze multiple data points over time. Each form of evaluation provides its own benefits.

Internal evaluation measures give students an important and central voice in programmatic improvement. Although we use our professional judg- ment when making program revisions, it is essential to hear directly from candidates themselves how they experience those decisions. Their ongoing written and oral feedback provides an important window into their thinking. For instance, we feel strongly that Alfie Kohn’s notion of working “with” children rather than doing “to” them is essential to establishing a classroom learning community. While the interns appreciated Kohn’s ideas, we learned during an exit focus group interview that implementing his ideas from the outset of their internship was challenging. This insight led us to revamp our approach to teaching classroom management, including moving our exami- nation of Alfie Kohn’s ideas to the end of their internship rather than keeping it at the beginning.

A second benefit to internal evaluation is that it fuels our efforts to reflect on and improve our own practice. A major premise of our teacher prepara- tion program is that good teachers are students—of children and childhood, learners and learning, curriculum and pedagogy. In other words, teaching is “the learning profession” (Darling-Hammond & Sykes, 1999) where contin- ual inquiry is central to a teacher’s personal and professional growth over a lifetime of practice (Barth, 1990). We want our teacher candidates to become reflective practitioners, routinely engaging in the analysis of their teaching and students’ learning. We deliberately model how we ourselves make sense of our own pedagogy and their learning, including how we collect and process their feedback— daily/weekly/over time— to make programmatic changes. As discussed earlier, a case in point involves the exit focus group interview that we hold with interns. We elicit their thoughts about strengths of the current program and suggestions for improvement while also shar- ing changes that we are already considering making based on their ongoing

Using Internal and External Evaluation 19

feedback throughout the year. We then ask them to consider whether/how those proposed changes make sense given their experience in the program.

A third benefit to internal evaluation is that it provides space for and validates our own professional judgment. All of the weekly Checking Ins, the weekly observations and debriefings of interns’ classroom practice, the end-of-class written reflections, the continuous email and face to face con- versations and consultations that we hold with them continuously inform whether/how we are creating educative learning opportunities for candi- dates. In this sense, just as we want our own candidates to feel that they are doing “with” us rather than being done “to” in the program, we appreciate having a voice in determining whether/how to respond to data we believe is important to collect and analyze, not just the data points that we are required to collect and analyze for national accreditation. We have become so data driven in education that it is easy to feel that only numbers matter; in doing so, however, we ignore at our peril educators’ direct knowledge of students drawn from years of practice and routinely conducted informal assessments. These less formal assessment arenas are central to engaging in program improvement because they address aspects of teacher candidates’ teaching and learning that are not easily quantifiable.

That said external evaluation has its own benefits. NCATE provides a second set of eyes and a broader group of colleagues to think about and analyze our program’s success and progress. When analyzing one’s own pro- gram, it can be challenging to identify blind spots: “When you are already familiar with a culture or group or school, your angles of vision are nar- rowed by preformed assumptions about what is going on” (Glesne, 1999, p. 25). External evaluation measures force us to think outside of our own context and to consider concerns that are not otherwise apparent as an insider. For instance, we never would have considered making the Teaching for Understanding entries stand-alone assessments had the external concern not been raised that we were not demonstrating that our candidates possess content-specific planning, teaching and assessment skills. Doing so, how- ever, created greater coherence with the way the program is structured. Assignments now have a meaningful time and purpose within the program. In addition, the Teaching for Understanding entries are more rigorous now that there is greater clarity with what a high-quality practice looks and sounds like in a given content area. Moreover, the intern receives specific feedback on her own planning, teaching and assessment efforts in relationship to that quality.

External evaluation also allows us to model to our teacher candidates how to approach high-stakes testing measures. We want them to see the role that state standards and assessments, along with other forms of data, can play in informing their instructional decision-making. In this data driven world, it is not just about collecting required data but also using those data to inform decisions about curriculum and instruction. We have to actively

20 P. J. Norman and S. A. S. Sherwood

seek out external feedback and make sense of it, remaining true to our own values/beliefs while also being responsive to concerns raised. We have been transparent with them how we have processed NCATE feedback and used it to make programmatic improvements, pointing out that this process parallels what they must do as classroom teachers in light of standards-based student achievement data.

Finally, external evaluation measures hold teacher educators account- able. In order to meet requirements of national accreditation, a program must demonstrate that it has done what it said it was going to do. Making program results public means that we must move from a stance of “this is what I hope the program is” to “this is what the program is” by providing concrete outcome-based evidence to support those claims.

While we advocate a model of program improvement that embraces both internal and external evaluation components, such a model is not with- out challenge. Although both forms of evaluation increase rigor in our pro- grammatic decision-making, they also create enormous stress on students to complete all formal and informal assessments. On the one hand, the time that our teacher candidates spend documenting and demonstrating the quality of their teaching and children’s learning creates both important reflection points for candidates and important data points for program evaluation. On the other hand, that writing and reflecting time means less time for candidates’ actual classroom instruction. Determining the most beneficial combination of classroom teaching and documentation of instructional performance remains an ongoing tension that we envision we must learn how to manage rather than resolve over time. This tension reflects the ever-present difficulty that classroom teachers face. Because of increased accountability measures, so much time is devoted to administering tests that there is less time to actually instruct students. It is also equally challenging to find time to use the results of such measures to inform future instruction in meaningful ways.

As teacher educators, we struggle to score all of the required assess- ments, both those needed for national accreditation and those that we have deemed important for our candidates’ growth, in a timely fashion. Keeping the administrative work manageable is another challenge.

We strive to ensure that our program improvement model informs our practice rather than becomes it. As labor intensive as the model is, both our professional experience and our external accreditor tells us that our candidates leave our program with the knowledge, skills and dispositions needed to plan, teach and assess elementary students’ learning effectively.

IMPLICATIONS

Even though our model of program improvement grows out of our work in a small teacher preparation context, larger institutions can experience

Using Internal and External Evaluation 21

many of the same benefits from combining internal and external evalua- tion efforts. Using an external accrediting body in concert with collecting formative assessment data from students and faculty insure that program improvement is responsive to the voices of both candidates and faculty while also addressing external organization standards. Many of the tensions that we face will undoubtedly surface for larger teacher preparation institutions as well. Any institution, regardless of size, must attempt to balance teacher can- didates’ clinical experiences with the time needed to analyze and write about those experiences as part of formal and informal data points. Managing the ongoing stress that both students and faculty face in generating and assessing those data points is another shared tension.

That said we imagine that the size of larger schools and departments of education also create unique challenges. In our program, our internal and external coordination efforts involve less than one dozen faculty and staff. In larger programs, a significant challenge lies in using internal and external assessment efforts to create shared understandings and program- matic changes beyond any individual educator’s practice or coursework. Coordination efforts not only need to address the collaboration of multi- ple instructors within a single course but also engage faculty across the entire teacher preparation curriculum to ensure that over time, the curriculum reflects teacher educators’ evolving understanding of the alignment between program goals, coursework, and learning outcomes for candidates as well as the ongoing feedback that candidates and external evaluators provide.

Larger teacher preparation programs must help all stakeholders develop a shared vision for teaching and learning that undergird the program. They must also construct shared understandings of the knowledge, skills and dis- positions that candidates must demonstrate all while knowing how any given component of the program enables those learning outcomes. Given the sheer number of faculty involved in larger teacher preparation programs, a chal- lenge lies in developing that shared vision and assessment system without becoming overly prescriptive about how teacher educators teach. In the same way that we want to avoid creating teacher candidates who simply become implementers of what someone else has declared that they should do to students, leaders within large teacher preparation programs must grap- ple with the question of how to avoid de-professionalizing their own teacher educators.

REFERENCES

Barth, R. (1990). Improving schools from within: Teachers, parents, and principals can make the difference. San Francisco, CA: Jossey-Bass.

Brouwer, N., & Korthagen, F. (2005). Can teacher education make a difference? American Educational Research Journal, 42(1), 153–224.

22 P. J. Norman and S. A. S. Sherwood

Charney, R. (2002). Teaching children to care: Classroom management for ethical and academic growth. Turner Falls, MA: Northeast Foundation for Children.

Curtis, D., Bordelon, K., & Teitelbaum, K. (2010). Keep a focus on meaningful reform efforts instead of political agendas. Planning and Changing, 41(3/4), 133–146.

Darling-Hammond, L. & Sykes, G. (Eds.). (1999). Teaching as the learning profession: Handbook of policy and practice. San Francisco, CA: Jossey-Bass.

Dhonau, S. (2007). Creating a culture for the preparation of an ACTFL/NCATE Program Review. Foreign Language Annals, 40(2), 247–259.

Dillon, E., & Silva, E. (2011). Grading the teachers’ teachers: Higher education comes under scrutiny. Phi Delta Kappan, 93(1), 54–58.

Duncan, A. (2009, October). Teacher Preparation: Reforming the Uncertain Profession. Speech presented at Teachers College, Columbia University, N.Y. Retrieved from http://www.ed.gov/news/speeches/teacher-preparation- reforming-uncertain-profession

Fein, A. (2004). What’s good for the goose . . . Inconsistency in the NCATE Standards. Multicultural Education, 11(4), 54–55.

Greenberg, J., & Walsh, K. (2010). Evaluating the Fundamentals of Teacher Training in Texas. Washington DC: National Council on Teacher Quality. Retrieved from http://www.nctq.org/edschoolreports/texas/docs/report.pdf

Hart, L. C. (2002). Preservice teachers’ beliefs and practice after participating in an integrated content/method course. School Science and Mathematics, 102(1), 1–14.

Kohn, A. (2006). Beyond discipline: From compliance to community 2nd edition. Alexandria, VA: Association for Supervision and Curriculum Development.

Levine, A. (2006). Educating school teachers. Washington, DC: The Education Schools Project. Retrieved from http://www.edschools.org/pdf/Educating_ Teachers_Report.pdf

Levin, B. (2003). Answering the “So what?” question: What do these case studies tell us? Case studies of teacher development: An in-depth look at how thinking about pedagogy develops over time (pp. 233–299). Mahwah, NJ: Lawrence Erlbaum Associates.

Meyer, C. (2010). Program Assessment: Ways to make it meaningful while waiting for NCATE. National Social Science Journal, 34(2), 105–109.

Marzano, R.J. (2010). Formative assessment and standards-based grading. Bloomington, IN: Marzano Research Laboratory.

National Council for the Accreditation of Teacher Education (NCATE). (2013). Guidelines on assessment. Retrieved from http://www.ncate.org/Accreditation/ ProgramReview/GuidelinesAndProcedures/GuidelinesonAssessment/tabid/446/ Default.aspx

National Council for the Accreditation of Teacher Education (NCATE). (2008). Professional Standards for the Accreditation of Teacher Preparation Institutions. Washington DC: Author. Retrieved from http://www.ncate.org/Portals/0/ documents/Standards/NCATE%20Standards%202008.pdf

No Child Left Behind Act of 2001, 20 U.S.C. § 6319 (2008). Retrieved from http:// www2.ed.gov/policy/elsec/leg/esea02/index.html

Using Internal and External Evaluation 23

Washburn, E.K., Joshi, R.M., & Cantrell, E.B. (2011). Are preservice teachers prepared to teach struggling readers? Annals of Dyslexia, 61, 21–43.

Watson, M. (2003). Learning to trust: Transforming difficult elementary classrooms through developmental discipline. San Francisco, CA: Jossey-Bass.

Yao, Y., & Williams, W. (2010). A comparison of three teacher preparation programs in terms of their impact on teacher competencies. Action in Teacher Education, 32(3), 15–24.