Summer End 2

Program Evaluation Proposal 1

An evaluation of the doctoral program in applied research psychology

Psychology 3512 Program Evaluation

Anonymous

Program Evaluation Proposal 2

An evaluation of the doctoral program in applied research psychology

This proposal presents an evaluation plan for the doctoral program in Psychology

currently offered at the University of Texas at El Paso (UTEP). The Doctor of Philosophy

(Ph.D.) program at UTEP is designed to produce bilingual/bicultural research psychologists that

will be able to serve Hispanic populations in Texas. The Ph.D. program at UTEP aims “to

prepare research psychologists to address questions applicable to the English-Spanish bicultural

communities in the Southwest” (The University of Texas at El Paso, 1992, p. 8). Though the

program is fairly new (September of 1993), its development can be traced back to the late 1970s

when the faculty first began to initiate efforts to pursue a Ph.D. program for the department (J. V.

Devine, personal communication, March 27, 1998). Since its development the program has

enjoyed support from the University and the El Paso community.

Program Overview

The program aims to provide the local community, the state and the nation with trained

bilingual/bicultural research psychologists. In fact, there is no other program available to train

bilingual/bicultural research psychologists in the nation (Jones, Keppel & Meissen, 1993). The

Ph.D. program at UTEP aims to provide local residents from both the El Paso and Juarez

community with the opportunity to pursue Ph.D-level training in psychology. Though there are

other Ph.D. programs in the State of Texas, there is no Ph.D. programs--other than that currently

offered by UTEP department of psychology--available to the local community. Thus, the Ph.D.

program will enable the local community to pursue Ph.D. level training and provide access to

and retain trained bilingual/bicultural psychologists. At the state level, the Ph.D. program aims

to increase the number of psychologist trained to work with Hispanic populations in Texas. This

particularly important to Texas because it is among the states with higher percentages of

Program Evaluation Proposal 3

Hispanics in their population. The Ph.D. program at UTEP can serve to provide trained

psychologists to deal with issues faced by Hispanics living in Texas. Nationally, the Ph.D.

program will offer specialized training in working with Hispanic populations. This is important

since the Hispanic population is rapidly growing and is emerging as a significant minority

population in the United States (Marin & Marin, 1991). Thus, the Ph.D. program will serve an

important and emergent need for trained bilingual/bicultural research psychologists nationwide.

Program Description & Objectives

To meet its objectives the program emphasizes the application of research methods and

findings in non-academic settings. This includes a core curriculum in experimental psychology

along with a field placement and specialization courses in either health or human behavior in

organizations. The program assumes that preparation for applied psychologists requires a firm

foundation in general experimental psychology including courses in statistics, cross-cultural

research, and field research methodology as applied to bilingual/bicultural settings. In addition,

the program requires its students to undertake a field placement designed to meet the goals of the

program and the needs of the individual student. The field placement is the cornerstone of the

program and is designed “to provide the practical experience in an organizational setting that will

enhance the student’s training as an applied research psychologist”(The University of Texas at El

Paso, 1992, p.16). Together with the academic training, the internship will provide students with

a well rounded education and enable them to function as bilingual/bicultural psychologists.

To be eligible for the program an applicant must have (a) a B.A. from an accredited

university, (b) a minimum 3.0 G.P.A. in undergraduate work, (c) a minimum of 3.0 G.P.A. in

their psychology course work, (d) a minimum score of 500 on each of the subtests of the GRE

test, (e) satisfactory background in statistics/experimental work, and (f) three letters of

Program Evaluation Proposal 4

recommendation. In addition, international students must also (g) have satisfactory scores in the

TOEFL exam. This criteria is similar to that used by most other graduate programs in

psychology in the US and Canada (American Psychological Association, 1994; Norcross,

Hanych, & Terranova, 1996; Purdy, Reinehr, & Swartz, 1989).

In sum, the purpose of the Ph.D. program at UTEP is “to prepare research psychologist to

address questions applicable to the English-Spanish bicultural communities in the Southwest”

(The University of Texas at El Paso, 1992, p. 8). This program is based on the notion that there

is an increasing need for trained bilingual/bicultural psychologists to handle the needs of the

Hispanic community at the local, state and national levels (See Table 1). Futhermore, there is no

equivalent program available in El Paso, or in the State of Texas. This program is of timely

importance since the Hispanic population is becoming a significant minority group in the US

(Marin & Marin, 1991) and their growth is likely to have a significant impact on the work force

as well as other areas. This program assumes that by providing specialized training it will be

able to prepare bilingual/bicultural research psychologists.

Purpose of the Evaluation

This proposal presents an evaluation plan for assessing the progress of the Psychology

Ph.D. program at UTEP. At the outset it must be recognized that this evaluation plan comprises

a formative rather than a summative evaluation. This approach was followed for several reasons.

First, the Ph.D. program at UTEP has been in place for approximately five years. Given that on

average most Ph.D. candidates complete their training in 6 years it is unreasonable to expect that

the Ph.D. program will have met its objectives during its fifth year of implementation. Secondly,

the Ph.D. program is the first of its kind for the department of psychology and as a result will

undergo procedural changes commonly associated with the development of new programs in any

Program Evaluation Proposal 5

organization. Finally, this evaluation seeks to provide evaluation information that can be utilized

to improve the program rather than assess its absolute merit or worth at this point. Accordingly,

this evaluation aims to provide information that can be utilized by program administrators,

faculty, staff and students of the UTEP doctoral program in psychology to improve the program

and address issues of concerns.

Evaluation Questions

Specifically, this evaluation proposes to examine the following evaluation questions: (1)

To what extent has the program been implemented as proposed? (2) To what extent has the

program progressed toward meeting its stated objective of providing trained “bilingual/bicultural

research psychologists”? (3) What are the perceptions of the various stakeholder regarding the

Ph.D. program? and (4) What are the main concerns for program development and improvement

among the faculty and students of the program? These evaluation questions have been selected

to represent broad areas of interests that all stakeholders (e.g., local staff and faculty, students in

the program as well as higher level university and public officials) share in common. In

addition, these questions can provide valuable information that can be directly fed back to the

program as well as enable the participants of the program to voice issues of concerns and make a

worthwhile contribution toward the improvement of the graduate education offered at UTEP.

Evaluation Design

This evaluation concerns both process and implementation issues. That is, this evaluation

will assess whether the Ph.D. program has been implemented as proposed and whether the

program has made progress in that process. Shortell & Richardson (1978) have argued that

“recurrent institutional cycle designs provide a flexible way of handling some major threats to

internal validity and is particularly conducive to settings and circumstances in which new groups

Program Evaluation Proposal 6

of individuals are exposed to ongoing programs” (p. 63). The Ph.D. program at UTEP certainly

would fit this description. The UTEP program has been in place for approximately five years

and has had a new cohort of students entering the program each year since it began. This design

allows for the integration of data from multiple sources (e.g., program participants at various

stages of the programs, faculty, and staff) while allowing for diversity in information. This

design is made up of three designs including the one-shot case study, a static group comparison

design and the single group pre-posttest design. This design offers several advantages in that it

controls for the effects of history, testing , instrumentation, selection and attrition. However, it

does not rule out the effects of maturation or regression. Because this evaluation is not

concerned with generalizing outside of UTEP at this time, external validity issues (e.g.,

generalizing to other participants, settings, times) are not of major concern.

Population and Sampling Process

This evaluation will be conducted with the program participants at the University of

Texas at El Paso. The proposed evaluation will include both paper and pencil assessments of

participants satisfaction with the program, supervision, coursework, research and other areas of

the program. In addition, focus group interviews will be conducted with each student cohort

enrolled since the program was installed as well as with both the faculty and staff of the program.

Focus group interviews will be carried in one of the seminar rooms of the department. These

rooms are generally medium in size and can accommodate up to 20 people. As recommended by

focus group researchers (Krueger, 1994), the sessions will be tape-recorded to ensure the

accuracy of the data.

Participants. Participants will include students, faculty and staff of the Ph.D. program at

the University of Texas at El paso. Participant will be asked to complete both a written survey

Program Evaluation Proposal 7

and a focus group interview designed to assess their perceptions and attitudes about the Ph. D.

program at UTEP. In addition, individual interviews will be conducted with various university

officials including the Dean of the Liberal Arts College, University Vice President of Academic

affairs and the UTEP President to assess their knowledge about the programs’ success.

Individual interviews will also be conducted with other University officials that are presently

involved with the program or were involved in its formulation, or development.

Measures

Participant survey. Participant perceptions about the program will be assessed through a

survey questionnaire (see Appendix A). This survey will assess various aspects of the program

including participants knowledge, development and satisfaction (e.g., satisfaction with the

program, supervision).

Participant knowledge. Participants’ knowledge about the program will be assessed with

a scale specifically designed for this project. The scale is comprised of four items that are

presented in Likert format. Participants rate the extent to which they agree or disagree with each

item on a scale ranging from one to five. Scoring is accomplished by summing across the four

items. With a five-point response scale, scores can range from a low of 4 to a high of 20. Higher

scores on the scale indicate greater knowledge about the program.

Development. Participant development will be measured using a scale designed

specifically for this evaluation. The scale is comprised of 6 items that are presented in Likert

format. Participants rate the extent to which they agree or disagree with each item on a scale

ranging from one to five. Scoring is accomplished by summing across the 6 items. With a five-

point response scale, scores can range from a low of 6 to a high of 30. Higher scores on the scale

indicate more positive views about participant development in the program.

Program Evaluation Proposal 8

Satisfaction. Satisfaction with the program, supervision, co-workers will be measured

using a scale modelled after the Job Descriptive Index (JDI) developed by Smith, Kendall, &

Hulin (1969). Each scale contains an 18-item index that presents respondents with adjective

checklists. Respondents indicate "yes" if the adjective describes their work (or co-worker,

supervision, pay), "no" if it does not describe their work (or co-worker, supervision, pay), and

"?" if they cannot decide.

Focus groups. Focus group interviews will be administered to individual stakeholder

groups (see Appendix B). The focus group interview is composed of four parts. Part I is

designed to determine the participant's attitudes and feelings toward program. It contains three

questions dealing with the perception about the Ph.D. program (e.g., What is your general

perception of the program? What do you think of it?). Part II is designed to explore participant's

knowledge of the program. It contains several questions that assess the participant's knowledge

and experience with the program (e.g., What do you perceive as the purposes or guiding

philosophy of the program?). Part III and IV are designed to understand participant's concerns

about the program. It contains questions that deal with participant's feelings about the program,

its objectives, goals and overall purposes (e.g., What are some concerns that you have about the

program?).

In addition, the present evaluation will also collect archival data to assess the programs

progress. Archival data collected will include samples of students program plans for both

current and graduated students. These data will be used to examine the extent to which the

program has been implemented as proposed.

Procedures

Participants will be asked to attend a discussion session with the program evaluator. The

Program Evaluation Proposal 9

participants will be scheduled to attend their session with their respective cohort (e.g., 1st. year

students, faculty, staff). At the begining of the session the evaluator will give a brief

introduction and describe the purpose of the session in general terms. Participants will be

informed of their rights as participants. They will be told that their responses are anonymous and

only the evaluator will have access to their individual responses. Each participant will then be

given a questionnaire to complete and place into a box located near the entrance of the room.

When the group is finished, the evaluator will then begin the focus group discussion with the

group.

Focus group interviews will be conducted in one of the seminar rooms in the department

of psychology building by the evaluator and an assistant. Again, participants will be informed of

their rights and they will be told that their responses are to be kept anonymous. Participants will

be told not to state their name or any other identifying information during the focus group

interviews. Approximately 60-90 minutes will be required to complete the focus group

interviews. However, should more time be needed it will be determined at the time of the

interview with the consent of all of the participants. Upon completion of the interview,

participants will be given the opportunity to ask questions regarding the purpose of the

evaluation. All responses and comments about the interview will be recorded. The tape recorder

will be turned off by the evaluator when the last participant has left the room.

Analysis

Appendix C outlines the evaluation questions along with the information required for

each question, the information source and the method of data collection. As can be seen the

evaluation questions will be answered using a combination of quantitative and qualitative

information. For the quantitative data reliability indices will be calculated to determine the

Program Evaluation Proposal 10

internal consistency of the items for all the scales included in the questionnaire. Scores on each

of the scales will be used to compare across participant groups as well as to the narrative

responses from the focus group interviews.

Qualitative survey comments and focus group interview responses will be content

analyzed and major themes will be identified for the entire group (e.g., including all participants)

as well as for each group separately (e.g., student responses, faculty responses etc.). The data

will be coded and analyzed by two independent judges to cross check the reliability of the coding

procedures. Major themes and concerns will be compiled for comparisons between the

stakeholder groups.

Personnel, Equipment & Logistics

The evaluation design described in the design section, will be conducted with the

program participants at the University of Texas at El Paso. The university is located near the

Mexico-US border approximately 3-5 miles from Ciudad Juarez, Mexico. It is a midsize

university with a predominantly Hispanic student population from the surrounding communities.

Both the participant survey and focus group interviews will be carried in one of the

seminar rooms of the department (e.g., Psych. 308). These rooms are generally medium in size

and can accomodate up to 20 people. As recommended by focus group researchers (Krueger,

1994), the sessions will be tape-recorded to ensure the accuracy of the data. Participants will be

asked to attend a discussion session with the program evaluator. The participants will be

scheduled to attend their session with their respective cohort (e.g., 1st. year students, faculty,

staff).

Evaluation time line. Approximately seven months will be needed to conduct the full

evaluation. This time will be used to develop measures, conduct observations, administer the

Program Evaluation Proposal 11

survey, analyze data, review with clients and to prepare and present a final report. A significant

portion of time will dedicated to the gathering of focus group interviews (e.g., 2 months) as well

as to the analysis of this data. All other aspects of the evaluation will last no longer than one

month. Although some personnel in the department of psychology may be qualified to assist in

the evaluation, because of the nature of the data being collected and confidentiality issues the

expertise of individuals in the department will not be utilized.

Implications & Recommendations

The results from both the questionnaire survey and the focus group interview will provide

data on stakeholders assessment of the Ph. D program at UTEP. The result will help determine

whether the program has made progress toward achieving its goals. Furthermore, the findings

will also help to identify program strengths and weaknesses. Specifically, the findings of the

proposed evaluation will provide information on (1) the extent to which the program has been

implemented as proposed; (2) the extent to which the program has made progress toward

meeting its stated objectives; (3) provided information about the perceptions of stakeholders

regarding the Ph.D. program; and (4) help identify areas of concern for the purposes of program

development and improvement. Furthermore, this evaluation plan can provide information that

can guide the development of similar programs in other schools. This is particularly important

since the strength of the Hispanic community is rapidly increasing and the need for such

programs may become acute.

Human Subjects

The proposed evaluation plan aims to assess the Ph. D. program at UTEP. Participants in

this project will be required to complete an anonymous survey assessing their perceptions about

the Ph.D. program as well as a focus group interview with other participants. Participants will

Program Evaluation Proposal 12

include students, faculty and staff as well as other selected university and public officials

associated with the program. Local stakeholders (e.g., students, faculty, staff and other UTEP

personnel) will be asked to participate in a discussion with the program evaluator. A flyer with a

brief description of the project will be placed in individual mailboxes as well as throughout

various information boards in the department. The flyer will include a brief description of the

project, as well as the date, time, and location of the study.

Questionnaires will be administered to participants on the same day as the scheduled

focus group discussions. Participants will be informed of their rights as participants. They will

told that responses are anonymous and only the evaluator will see the completed questionnaires.

Instructions will be read aloud and participants will be given a pencil, an informed consent form,

the survey questionnaire. Participants will be told not to write their name or any other

identifying information on the survey itself. Once completed the participants will place their

individual surveys into a box located near the entrance of the room. After a brief break, focus

group discussion will begin.

Focus group interviews will be conducted in one of the seminar rooms in the department

of psychology building by the evaluator and an assistant. Again, participants will be informed of

their rights. Participants will be told not to state their name or any other identifying information

during the focus group interviews. Approximately 45-60 minutes will be required to complete

the focus group interviews. Upon completion of the interview, participants will be given the

opportunity to ask questions regarding the purpose of the evaluation. All responses and

comments about the interview will be recorded. The tape recorder will be turned off by the

evaluator when the last participant has left the room.

Program Evaluation Proposal 13

References

American Psychological Association. (1994). Graduate study in psychology. Washington, DC:

Author.

Jones, J. M., Keppel, G., & Meissen, G. (1993, February). Site visit report: Department of

Psychology University of Texas at El Paso. Berkeley, CA.

Krueger, R. A. (1994). Focus groups (2nd. ed.). Thousand Oaks, CA: Sage.

Marin, G., & Marin B. V. (1991). Research with Hispanic populations. Newbury Park, CA.

Sage.

Norcross, J. C., Hanych, J. M., & Terranova, R. D. (1996). Graduate study in psychology: 1992-

1993. American Psychologist, 51, 631-643.

Purdy, J.E., Reinehr, R.C., & Swartz, J.D. (1989). Graduate admissions criteria of leading

psychology departments. American Psychologist, 44, 960-961.

Ronco, S., Passmore, B., Morales, T., & Schwartz, O. (1997). The University of Texas at El

Paso fact book 1996-1997. El Paso, Texas. The University of Texas, Office of

Institutional Studies.

Shortell, S. M., & Richardson, W. C. (1978). Health program evaluation. Saint Louis, MO. The

C. V. Mosby Company.

Smith, P. C., Kendall, L. M., & Hulin, C. L. (1969). The measurement of satisfaction in work

and retirement. Chicago, IL. Rand Mc Nally.

The University of Texas at El Paso (1992). Doctor of philosophy in psychology. El Paso, Texas.

University of Texas

Program Evaluation Proposal 14

Table 1. Process Model of Evaluation.

Preexisting Conditions Program Components Intervening Events Impact/Consequences

No Ph.D. programs

exist to meet the needs

of the Hispanic

Population

Ph.D. programs in

Texas are beyond their

estimated capacity to

train additional

psychologist for the

state.

There is no Ph.D.

program available to the

El Paso-Juarez

community.

-Inputs/Obj.

-Prepare research

psychologist to serve

the English-Spanish

Bicultural

communities in the

Southwest.

Resources

-Graduate Faculty

-Adequate facilities

-Adm. Support

-Library resources

-Computing resources

-Extramural funding

Activities

-External evaluations

have been conducted.

-Recruitment of students

has continued.

Internal

-Faculty Turnover

-Budgetary changes

-Student dropout rate

-Availability of

assistantships

-Student progress

External

-Extramural funding

-Poor applicant pool

-Low visibility

Proximal

-increased pool of

qualified applicants

-increased enrollment

in the Ph.D. program

-increases in

applicants from the El

Paso/Juarez area

-increased fluency in

Spanish language

among students

-increases in the

number of students

passing the language

proficiency exam.

-increases in the

passing rate of

students taking the

first year exam

-increases in the rate

of students reaching

ABD

-increases in

extramural funding

awards to faculty

Distal

-Increase the number

of bilingual/bicultural

applied research

psychologists

-Greater cooperation

/communication

between UTEP and El

Paso-Juarez

community

-Increase KSAs of

Psychologist

Program Evaluation Proposal 15

APPENDIX A

Participant Questionnaire

Program Evaluation Proposal 16

Survey Questionnaire

We are interested in learning how to improve the Ph.D. program at UTEP. Please give your honest opinions to the questions asked here. Information collected on this survey will assist in the evaluation of the Ph.D. program and will be used to make improvements. All information is confidential and only the program evaluator will have access to the information you provide--you as an individual will be anonymous.

Directions: Read each statement carefully and check the box that best represents your opinion next to each statement.

Participant’s Knowledge Strongly Disagree

Disagree Neither Agree Nor Disagree

Agree Strongly Agree

There are clear guidelines on the program requirements. I am informed of program changes affecting my educational goals

Information is shared with students is timely and organized. There is adequate information/resources available to students.

Participant Development Strongly Disagree

Disagree Neither Agree Nor Disagree

Agree Strongly Agree

Student research is taken seriously in the department. I would like to influence decisions impacting students.

I find the coursework in the program worthwhile. I find the coursework in the program useful.

There is good cooperation among faculty and students. Recently, I have considered leaving the Ph. D. Program.

Directions: Read each statement carefully and check the box that best represents your opinion next to each statement.

PROGRAM IN GENERAL Think of the Ph.D. program at UTEP. How well do the words or phrases describe the program in general?

Yes No Don’t Know

CURRENT POSITION Think of the work you do at present in the Ph.D. program. How well do the words or phrases describe what your position is like most of the time?

Yes No Don’t Know

Pleasant Fascinating Bad Routine

Ideal Satisfying Waste of time Boring

Good Good Undesirable Creative

Worthwhile Respected Worse than most Uncomfortable

Acceptable Pleasant Superior Useful

Better than most Tiring Disagreeable Healthful

Makes me content Challenging Inadequate Too much to do

Excellent Frustrating Rotten Simple

Enjoyable Repetitive Poor Gives sense of accomplishment

Program Evaluation Proposal 17

Directions: Read each statement carefully and check the box that best represents your opinion next to each statement.

SUPERVISION

Think of the Ph.D. program. How well do the words or phrases describe the supervision you receive in the program?

Yes No Don’t Know

COWORKERS-PEERS

Think of the Ph.D. program. How well do the words or phrases describe your coworkers or peers?

Yes No Don’t Know

Asks my advice Stimulating Hard to please Boring

Impolite Slow Praises good work Helpful

Tactful Stupid Influential Responsible

Up-to-date Fast Doesn’t supervise enough Intelligent

Has favorites Easy to make enemies Tells me where I stand Talk too much

Annoying Smart Stubborn Lazy

Knows job well Unpleasant Bad Gossipy

Intelligent Active Poor planner Narrow interests

Around when needed Loyal Lazy Stubborn

Please use this space to give us your reactions and views of the Ph.D. program at UTEP.

1. Do you like the content of the program?  Yes  No

Why or why not? _________________________________________________________________________________________________

_________________________________________________________________________________________________

_________________________________________________________________________________________________

2.. Did you like the faculty of the program?  Yes  No

Why or why not? _________________________________________________________________________________________________

_________________________________________________________________________________________________

_________________________________________________________________________________________________

3. What did you like most about the training you have received thus far?

_________________________________________________________________________________________________

_________________________________________________________________________________________________

______________________________________________________________________________________________ ___

4. What do you like least about the training you have received thus far? _________________________________________________________________________________________________________________________

_________________________________________________________________________________________________

_________________________________________________________________________________________________

Program Evaluation Proposal 18 5. Are there any areas covered in the program that you feel should have been covered in more detail?

_________________________________________________________________________________________________

_________________________________________________________________________________________________

_________________________________________________________________________________________________

6. Would you like to see other courses added to the program?  Yes  No If YES, in what area? _________________________________________________________________________________________________

_________________________________________________________________________________________________

_________________________________________________________________________________________________

7. Of the courses you have taken so far, indicate those which have been the most useful: _________________________________________________________________________________________________

_________________________________________________________________________________________________

_________________________________________________________________________________________________

8. Of the courses you have taken so far, indicate those which have been the least useful: _________________________________________________________________________________________________

_________________________________________________________________________________________________

_________________________________________________________________________________________________

THIS INFORMATION IS CONFIDENTIAL AND WILL ONLY BE USED TO ANALYZE THE RESULTS. IT WILL HELP US MAKE IMPROVEMENTS ON THE TRAINING. DEMOGRAPHICS

Gender:  Male  Female

Student Status:

 Ph. D. Student

 Health Track  Human Behavior in Organization Track

 MA Student

 General Experimental  Clinical

Number of years in the program:

 Ph. D. Student

 1st. year.  2nd. year.  3rd. year.  4th. year.  5th. year.  6th. year.

 MA Student

 1st. year.  2nd. year.  3rd. year.  4th. year.  5th. year.  6th. year.

Program Evaluation Proposal 19

APPENDIX B

Focus group interviews

All of you have been asked to participate in this study because you can help me understand how

people like yourselves [e.g., students, faculty, staff, or other] feel about the Ph.D. program

offered by the department of psychology. We are interested in finding out what you feel about

the Ph.D. program--be that positive or negative. There are no wrong or right answers; we just

want to know what you think and feel about the program. Remember your answers are

completely private. No one except the evaluator (myself) will have access what you say today.

To begin with we are going to start by completing a brief survey that explores how you feel

about the program. We will begin our discussion as soon as that is completed. Take a few

moments and answer the survey then place it face down on your table so that I know when it is

alright to begin.[WAIT FOR PARTICIPANTS TO COMPLETE THE SURVEY].

Now I am going to turn on the tape recorder and check to make sure it is functioning properly. I

would like all of us to count to five to check the tape recorder, I will begin and the next person to

my right continue till we have gone full circle. I will then check to make sure the recorder is

functioning properly and begin the discussion. To keep your answers private, we will not use

your name or any other type of identifying label. [REWIND TAPE AND PLAY BACK TO

ENSURE THAT THE RECORDINGS ARE AUDIBLE].

Part I. Goal: Determine attitudes and feelings toward the Ph.D. program.

1. What is your general perception of the program?

--What do you think of it? Do you think well of it?

2. What do you like about it? What do you think the program does well?

--What makes you say that?

3. What do you think is bad about the program? What don’t you like about the program?

--What makes you say that? Explain ?

Part II. Goal: Explore participants knowledge of and about the program.

4. What do you perceive as the purposes (goals, objectives) or guiding philosophy of the

program?

--How did you arrive at this answer ?

--What kinds of things happened that told you this?

5. What do you think about this philosophy? Do you agree with these purposes or philosophy?

Do you think the problems the program addresses are severe? important?

Program Evaluation Proposal 20

6. I am going to read to you the stated objectives of the program as written in the “gray book”--

the program’s official plan that delineates the purpose and objectives.

Read [The Doctor of Philosophy (Ph.D.) program at UTEP is designed to

produce bilingual/bicultural research psychologists that will be able to serve

Hispanic populations in Texas. The Ph.D. program at UTEP aims “to prepare

research psychologists to address questions applicable to the English-Spanish

bicultural communities in the Southwest” (The University of Texas at El Paso,

1992, p. 8)].

7. What do you think about this philosophy? Do you agree with these purposes or philosophy?

Do you think the problems the program addresses are severe? important? Is it different from

what you expected? Why or why not?

8. What do you think the theory or model for the program is? Why/how do you think it works?

How is it supposed to work? Why would the program actions lead to sucess on the program’s

objectives or criteria? Which program components are most critical to success?

9. Typically, what types of activities do students in the department engage in? (academic

activities)?

10. Are these activities different from that of other students in other schools? Explore ?

11. What does a student have to do to finish the program?

Part III. Goal: Understand participant concerns about the Ph.D. program

13. What concerns do you have about the program? About its outcomes? Its operations? Other

issues?

--What makes you say that?

14 If there any changes that you would like to see occur what would they be?

How would these help you or others in the program?

Do you think others share your concerns?

Do you think these would be easily implemented?

15. Why do you think these changes have not occurred ?

Program Evaluation Proposal 21

Part IV. Goal: Learn about issues related to specific participant groups

Faculty

A. Is the program being implemented as originally conceptualized?

B. Are there any problems with the implementation process?

C. What problems do faculty perceive?

D. How satisfied are they with the way the program is implemented”

E. How satisfied are they with the quality of students being enrolled?

F. How satisfied are they with the graduating students?

G. How well does the program prepare its students for the job market?

H. How comparable is the curriculum to similar graduate programs (nationally, or regionally)?

Students

A. Is the program meeting their expectations? Why or why not?

B. What are some problem areas in the program:

1. Classes: Have classes been offered regularly? Which have and which have not? are there

any additional courses that you would like to see offered?

2. Internships? How many students have gone on internships? How satisfied were they with

their experiences? How relevant were the internships to their career/professional goals?

3. Funding? How are funding decisions made?

4. Teaching opportunities? How many have taught classes? How is the assignment of

courses made? What preparation do students receive to teach a course?

5. Other?

Staff

A. Has there been added work as a result of the Ph.D. program beyond what is possible with the

resources available to the department?

General

A. Has the program graduated the first class of students? How prepared are they for the market?

How well does the program prepare its students for the job-market?

B. How comparable is the curriculum to those of other graduate programs (nationally, or

regionally)?

C. Has the program met accreditation standards (p. 29)?

16. Are there any comments or suggestions that you want to add ?

Program Evaluation Proposal 22

APPENDIX C

Evaluation Plan Outline

Evaluation

Question

Information

Required

Source of

information

Method

1. To what extent

has the program

been implemented as

proposed?

Qualitative Participant interviews

Archival data

Focus group interviews

Departmental records

2. To what extent

has the program

progressed toward

meeting its stated

objectives?

Qualitative Participant interviews

Archival data

Focus group interviews

Departmental records

3. What are the

perceptions of

stakeholders

regarding the

program?

Quantitative/

Qualitative

Participant survey

Participant interviews

Participant survey

Focus group interviews

4. What are the

main concerns for

program

development and

imrovement?

Quantitative/

Qualitative

Quantitative/

Qualitative

Participant survey

Focus group interviews