education

How to Evaluate a Reading Program

Author(s): Sidney J. Rauch

Source: The Reading Teacher , Dec., 1970, Vol. 24, No. 3 (Dec., 1970), pp. 244-250

Published by: International Literacy Association and Wiley

Stable URL: https://www.jstor.org/stable/20196483

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected]. Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at https://about.jstor.org/terms

Wiley and International Literacy Association are collaborating with JSTOR to digitize, preserve and extend access to The Reading Teacher

This content downloaded from �������������131.95.109.31 on Mon, 21 Dec 2020 14:18:54 UTC�������������

All use subject to https://about.jstor.org/terms

Sidney Rauch is a Professor of Reading and Education at Hofstra University, Hempstead, Long Island, New York.

How to evaluate a reading program SIDNEY J. RAUCH

the purpose of evaluation is to take a comprehensive, unbiased and cooperative look at the reading program and to decide what modifications or changes, if any, should be made to improve the program. Evaluation involves value judgments. It is not a care fully controlled research study. Recommendations for improvement of the program must consider not only what should be done, but what can be done. Hemphill (1969) places evaluation studies within the framework of decision making rather than research. The administrator, after taking all facts from the evaluation study into consideration, then must weigh their relationship to its effect upon the community, staff, and tax structure. Hemphill (1969) lists six characteristics of school evaluations:

1. The problem is almost completely determined by the situation in which the study is conducted. Many people may be involved in its definition and, because of its complexity, the problem initially is difficult to define.

2. Precise hypotheses usually cannot be generated; rather, the task becomes one of testing generalizations from a variety of research studies, some of which are basically contradictory. There are many gaps which, in the absence of verified knowl edge, must be filled by reliance on judgment and experience.

3. Value judgments are made explicit in the selection and the definition of the problem as well as in the development and implementation of the procedures of the study.

4. The study is unique to a situation and seldom can be replicated, even approximately.

5. The data to be collected are heavily influenced if not deter mined by feasibility. Choices, when possible, reflect value judg ments of decision makers or of those who set policy. There are often large differences between data for which the collection is feasible and data which are of most value to the decision mak ers.

6. Only superficial control of a multitude of variables important to interpretation of results is possible. Randomization to eliminate the systematic effects of these variables is extremely difficult or impractical to accomplish.

Four major steps are necessary in the evaluation of school 244

This content downloaded from �������������131.95.109.31 on Mon, 21 Dec 2020 14:18:54 UTC�������������

All use subject to https://about.jstor.org/terms

rauch: How to evaluate a reading program 245

reading programs. These are clarification of the roles of the evalu ator, collection of data, analysis of data, and reporting of data. "A Checklist for the Evaluation of Reading Programs," (Rauch, 1968) is used as an overall guide. The five major categories cov ered in this checklist are the reading program, the administrative and supervisory staff, the teaching staff, the pupils, and the parents.

CLARIFICATION OF THE ROLES OF THE EVALUATORS

Since value judgments are involved, it is recommended that the evaluation be conducted by a team of two to four reading specialists from different parts of the country so that the various backgrounds and points of view can be brought to the program. This type of representative team is preferable to the selection of a number of specialists from a single university or college whose philosophy or viewpoints may be too similar.

It is strongly recommended that the team meet with admin istrators and teacher representatives prior to the actual evaluation to explain the purposes and procedures, and to answer any ques tions. It is most important that the anxieties of teachers (partic ularly as to classroom observations) be allayed. The evaluation concentrates on the reading program; it is not an evaluation of individual teachers. No names or ratings of teachers are to appear in the final report. The purpose of the evaluation is a constructive one, i.e., to make recommendations for improvement. It is not a criticism of individual teachers, though strengths and weaknesses of reading techniques will be listed. If the evaluation is to be successful, it must have the confidence and cooperation of all concerned. Any team approaching the program with the intent to downgrade it or pick it apart is doomed to failure. Thus, the first and probably the most important step for the evaluation team is to gain the confidence and support of the teachers. This can only be done by clarifying all objectives and procedures before the evaluation actually begins.

COLLECTION OF DATA

Despite the various criticisms directed at standardized tests, test scores still remain an important part of reading evaluation. Robinson and Rauch (1965) describe the merits of standardized tests as follows:

Like other tools of teaching, standardized tests can be appraised in terms of both their form and their results. In their form?that is, their structure and operation?these tests have very important ad vantages: 1] their content is usually determined by careful design; 2] there are often parallel forms for comparison; 3] they permit

This content downloaded from �������������131.95.109.31 on Mon, 21 Dec 2020 14:18:54 UTC�������������

All use subject to https://about.jstor.org/terms

246 THE READING TEACHER Volume 24, No. 3 December 1970

many children to be treated simultaneously; and 4] they are ob jective in administering and scoring. In many schools, standard ized tests are the first step in identifying those students who are below grade level and who are in need of further diagnosis. They are particularly useful in measuring the wide range of reading levels in a class, school or school system. They also provide stand ards for comparing students on a nationwide basis. Standardized tests make a valuable contribution to modern education by dem onstrating rather clearly that children differ. They provide stand ards for making improvements in school programs in the areas of curriculum, school and classroom organization, and methods and materials of instruction.

A summary of cautions to be exercised in the use of stand ardized tests has been listed by Harmer (1967):

1. Test users should keep in mind that the test score is simply the result of performance on a particular day, at a particular time, and in a particular testing environment.

2. The diagnosis of reading achievement through the use of stand ardized tests may be fallacious unless carelessness and atti tudes toward taking tests on the part of the students are controlled.

3. The grade score on a standardized reading achievement test should not be thought of as precise indication of overall reading achievement; rather, it should be thought of as a measure of reading ability on that test at a particular point in time.

4. Teachers should keep in mind that grade scores on standard ized tests are derived by interpolating scores between grade levels or by extrapolating scores from one grade level to an other; thus, grade scores cannot be treated as empirically ob tained indications of month-to-month progress.

If these cautions are observed, and if schools begin develop ing their own local norms as the tests are used over a period of time, then the advantages of standardized tests will far outweigh their limitations. This point is reinforced by the American Psy chological Association in its Standards for Educational and Psycho logical Tests and Manuals: "Local norms are more important for

many uses of tests than are published norms." In addition to the evaluation of the results of standardized

reading tests, the results of individual and group IQ tests must be taken into consideration. Reading is primarily a mental process, and reading test results must be evaluated in terms of intellectual potential or capacity. For example, a different interpretation must be applied to the performance of two sixth grade classes whose median reading score places them both in the 60th percentile on a statewide basis, but one class has a median IQ score of 106 and the other a median IQ score of 118. Obviously, the latter group is not performing up to potential and the reasons for this discrepancy must be examined.

This content downloaded from �������������131.95.109.31 on Mon, 21 Dec 2020 14:18:54 UTC�������������

All use subject to https://about.jstor.org/terms

rauch: How to evaluate a reading program 247

Observation of teachers is the heart of reading evaluation. To evaluate, without observing teacher performance and pupil behav ior, is like reviewing a book without reading it. A representative daily schedule of the evaluator follows:

9:00 A.M.? 9:30 A.M. Observation of a first grade teacher. 9:30 A.M.?10:00 A.M. Conference with teacher following ob

servation. (Class is covered with by another teacher, administrator or substitute. )

10:00 A.M.?10:30 A.M. Observation of a third grade teacher. 10:30 A.M.?11:00 A.M. Conference with teacher following ob

servation. 11:00 A.M.?12 NOON Conference with one or two of the

following : administrator, reading specialist, librarian, school psycholo gist, school nurse, social worker, etc., (anyone connected with reading pro gram).

1:00 P.M.? 1:30 P.M. Observation of a sixth grade teacher. 1:30 P.M.? 2:00 P.M. Conference with teacher following ob

servation. 2:00 P.M.? 3:00 P.M. Evaluators confer, comparing notes

on observations and results of indi vidual conferences.

During the conference period following classroom observa tion, the topics covered should include: information about the class (e.g., reading range, potential, socio-economic background, interests), teacher's background and experience, her philosophy about reading instruction, and opinion about materials. It has been found most helpful to save the last five to ten minutes for the teacher's response to this type of question: "Assuming that budget problems were secondary, what recommendations would you make to improve the reading program in this school?"

Interviews with administrative personnel give them an op portunity to express their opinions about the reading program to the evaluator. In addition, each administrator is requested to sub

mit a brief statement in response to these three questions: 1] What are the strengths of the program? 2] What are the weak nesses of the program? and 3] What recommendations would you make to improve the program?

The author has used five opinion surveys as part of his eval uation of elementary reading programs. These are:

An Opinion Survey of Parents of Kindergarten Children, An Opinion Survey of Parents of Children, Grades 1-6, An Opinion Survey of Kindergarten Teachers,

This content downloaded from �������������131.95.109.31 on Mon, 21 Dec 2020 14:18:54 UTC�������������

All use subject to https://about.jstor.org/terms

248 THE READING TEACHER Volume 24, No. 3 December 1970

An Opinion Survey of Teachers, Grades 1-6, and Grade Level Observation Form (to be used by reading personnel).

The survey for teachers is included as an example.

An Opinion Survey of Teachers (Grades 1-6) Needs To

Strong Be Unable Program in General Component Good Fair Improved To Answer Basal Reading Program - -:? - - _ Word Recognition - - - - _ Comprehension - - - - - Critical Reading - -~ - - - Literature Program - - - - - Individualized Reading - - - - - Content Area Reading - - - - - Work and Study Habits - -- - - Supplementary Materials ?- - - - -

Meeting Needs of Children Meeting Individual Needs - - - - - Superior Reader - - - - - Average Reader - - - - - Disadvantaged Reader - - - - - Remedial Reader- - - Present Organizational Pattern - - - - - Diagnostic Services - - - - - Corrective Reading Program - - - - - Summer School Program - - - - - Supplementary Programs - - - - -

Helps to Classroom Teachers Administration - - - - - Consultants - - - - - In-Service Courses - - - - - School Meetings and Workshops - - - - - Materials - - - - -

Atmosphere Conducive to Learning Freedom to Develop Own Program - - - - - Development of Children's Love

of Reading - - - - - Competent Reading Leadership - - - - - Parent-Teacher Relationship - - - - -

It is important that all teachers be urged to complete their "Opinion Surveys" and turn them in to the evaluators. Names are not required. One reading program was harshly criticized by some members of the school board because only thirty-two out of sixty teachers had submitted their forms. The board's reaction, and rightly so, was "How can we have confidence in these teachers if they are not concerned enough to fill out a form which will help us in our evaluation?"

In summarizing the results of the parent opinion survey, make sure that two or more parents are involved in tabulation. Unless this is done, it is always possible that some parent (who is not too happy with the program) will question the accuracy of the findings.

This content downloaded from �������������131.95.109.31 on Mon, 21 Dec 2020 14:18:54 UTC�������������

All use subject to https://about.jstor.org/terms

rauch: How to evaluate a reading program 249

ANALYSIS OF DATA

It was previously noted that one hour at the end of each eval uation day (2:00 P.M.?3:00 P.M.) should be set aside for the evaluators to compare notes and opinions. These sessions also enable the evaluators to gather and classify data for the final report.

Each evaluator prepares his own individual report based on interviews, classroom observations, and analysis of opinion sur veys. The measurement specialist on the evaluation team is re sponsible for the analysis of standardized test scores. The director of the study analyzes the reports of each member of the evaluation team and prepares the final report.

In evaluating classroom performance, the following "Char acteristics of a Good Reading Lesson" are used as a guide: 1. Teacher has a definite goal or purpose for lesson and that

purpose is evident to students. 2. Lesson is planned, systematic, yet flexible according to dy

namics of classroom situation. 3. Classroom atmosphere is a pleasant, attractive and optimistic

one.

4. Attention is paid to individual differences. 5. Rapport between teachers and students is evident. 6. Teacher is diagnosing as she is teaching. 7. There is readiness for the lesson. 8. Pupils are motivated. 9. Materials are varied (basais, library books, workbooks, kits,

mimeographed materials, etc.). 10. Full use is made of audio-visual aids. 11. Questions are varied to check different levels of compre

hension. 12. Material is at appropriate level for students. 13. Teacher is obviously aware of such levels as "instructional,"

"independent," and "frustration." 14. Meaningful oral reading activities are used to check compre

hension. 15. Pupils have been trained in self-direction (i.e., go from one

activity to another without disturbing teacher). 16. All children are productively involved with some aspect of

reading. 17. Use is made of classroom and school libraries. 18. There is application of basic reading skills to content areas. 19. Efficient record keeping is done by teacher and students. 20. Teacher has sense of perspective and humor. 21. There is evidence of review and relationship to previously

learned material. 22. There are follow-up or enrichment activities.

This content downloaded from �������������131.95.109.31 on Mon, 21 Dec 2020 14:18:54 UTC�������������

All use subject to https://about.jstor.org/terms

250 THE READING TEACHER Volume 24, No. 3 December 1970

REPORTING OF DATA

Both an oral and written report of the findings of the evalu ation team are presented to the administrators' council, teacher representatives, and the school board. The school board, if it wishes, can then hold an open meeting for all parents and ques tions can be asked of the evaluators. Copies of the written report should be made available to all teachers. Three or four copies can be placed in the teachers' professional library or in a special sec tion of the school library.

A representative "Table of Contents" of the final written evaluation might follow this format:

A. Introduction (How the study was initiated) B. Description of the School-Community C. Characteristics of a Good Reading Program D. Evaluation Procedures (How the study was conducted) E. Strengths of the Program F. Weaknesses of the Program G. Recommendations (The heart of the report) H. Analysis of standardized test scores I. Results of Opinion Surveys (Appendix)

SUMMARY STATEMENT

There is a need for constant evaluation of reading programs. However, all concerned must participate. Teachers must have confidence in the evaluators, and the evaluators must recognize the many day-by-day problems faced by the average teacher. De spite the importance of standardized test results, the heart of the evaluation is classroom performance. Recommendations must be realistic. They must consider not only what should be done, but what can be done within a specific school-community environment.

In most instances, evaluation has a positive effect on the read ing program. It compels administrators and teachers to take a closer look at their methods, their materials, and their children? and this close examination generally results in progress.

REFERENCES Harmer, W. R. The selection and use of survey reading achievement tests. In

T. C. Barrett (Ed.) The evaluation of children's reading achievement. Perspectives in Reading, 1967, 8, 53-64.

Hemphill, J. K. The relationships between research and evaluation studies. In R. W. Tyler (Ed.) Educational evaluation: new roles, new means. The Sixty-eighth Yearbook of the National Society for the Study of Education, Part II, 1969, 189-220.

Rauch, S. J. A checklist for the evaluation of reading programs. The Reading Teacher, 1968, 21, 519-522.

Robinson, H. A., and Rauch, S. J. Guiding the reading program. Chicago: Science Research Associates, 1965.

This content downloaded from �������������131.95.109.31 on Mon, 21 Dec 2020 14:18:54 UTC�������������

All use subject to https://about.jstor.org/terms