Summer End 2
Program Evaluation Proposal 1
An evaluation of the doctoral program in applied research psychology
Psychology 3512 Program Evaluation
Anonymous
Program Evaluation Proposal 2
An evaluation of the doctoral program in applied research psychology
This proposal presents an evaluation plan for the doctoral program in Psychology
currently offered at the University of Texas at El Paso (UTEP). The Doctor of Philosophy
(Ph.D.) program at UTEP is designed to produce bilingual/bicultural research psychologists that
will be able to serve Hispanic populations in Texas. The Ph.D. program at UTEP aims “to
prepare research psychologists to address questions applicable to the English-Spanish bicultural
communities in the Southwest” (The University of Texas at El Paso, 1992, p. 8). Though the
program is fairly new (September of 1993), its development can be traced back to the late 1970s
when the faculty first began to initiate efforts to pursue a Ph.D. program for the department (J. V.
Devine, personal communication, March 27, 1998). Since its development the program has
enjoyed support from the University and the El Paso community.
Program Overview
The program aims to provide the local community, the state and the nation with trained
bilingual/bicultural research psychologists. In fact, there is no other program available to train
bilingual/bicultural research psychologists in the nation (Jones, Keppel & Meissen, 1993). The
Ph.D. program at UTEP aims to provide local residents from both the El Paso and Juarez
community with the opportunity to pursue Ph.D-level training in psychology. Though there are
other Ph.D. programs in the State of Texas, there is no Ph.D. programs--other than that currently
offered by UTEP department of psychology--available to the local community. Thus, the Ph.D.
program will enable the local community to pursue Ph.D. level training and provide access to
and retain trained bilingual/bicultural psychologists. At the state level, the Ph.D. program aims
to increase the number of psychologist trained to work with Hispanic populations in Texas. This
particularly important to Texas because it is among the states with higher percentages of
Program Evaluation Proposal 3
Hispanics in their population. The Ph.D. program at UTEP can serve to provide trained
psychologists to deal with issues faced by Hispanics living in Texas. Nationally, the Ph.D.
program will offer specialized training in working with Hispanic populations. This is important
since the Hispanic population is rapidly growing and is emerging as a significant minority
population in the United States (Marin & Marin, 1991). Thus, the Ph.D. program will serve an
important and emergent need for trained bilingual/bicultural research psychologists nationwide.
Program Description & Objectives
To meet its objectives the program emphasizes the application of research methods and
findings in non-academic settings. This includes a core curriculum in experimental psychology
along with a field placement and specialization courses in either health or human behavior in
organizations. The program assumes that preparation for applied psychologists requires a firm
foundation in general experimental psychology including courses in statistics, cross-cultural
research, and field research methodology as applied to bilingual/bicultural settings. In addition,
the program requires its students to undertake a field placement designed to meet the goals of the
program and the needs of the individual student. The field placement is the cornerstone of the
program and is designed “to provide the practical experience in an organizational setting that will
enhance the student’s training as an applied research psychologist”(The University of Texas at El
Paso, 1992, p.16). Together with the academic training, the internship will provide students with
a well rounded education and enable them to function as bilingual/bicultural psychologists.
To be eligible for the program an applicant must have (a) a B.A. from an accredited
university, (b) a minimum 3.0 G.P.A. in undergraduate work, (c) a minimum of 3.0 G.P.A. in
their psychology course work, (d) a minimum score of 500 on each of the subtests of the GRE
test, (e) satisfactory background in statistics/experimental work, and (f) three letters of
Program Evaluation Proposal 4
recommendation. In addition, international students must also (g) have satisfactory scores in the
TOEFL exam. This criteria is similar to that used by most other graduate programs in
psychology in the US and Canada (American Psychological Association, 1994; Norcross,
Hanych, & Terranova, 1996; Purdy, Reinehr, & Swartz, 1989).
In sum, the purpose of the Ph.D. program at UTEP is “to prepare research psychologist to
address questions applicable to the English-Spanish bicultural communities in the Southwest”
(The University of Texas at El Paso, 1992, p. 8). This program is based on the notion that there
is an increasing need for trained bilingual/bicultural psychologists to handle the needs of the
Hispanic community at the local, state and national levels (See Table 1). Futhermore, there is no
equivalent program available in El Paso, or in the State of Texas. This program is of timely
importance since the Hispanic population is becoming a significant minority group in the US
(Marin & Marin, 1991) and their growth is likely to have a significant impact on the work force
as well as other areas. This program assumes that by providing specialized training it will be
able to prepare bilingual/bicultural research psychologists.
Purpose of the Evaluation
This proposal presents an evaluation plan for assessing the progress of the Psychology
Ph.D. program at UTEP. At the outset it must be recognized that this evaluation plan comprises
a formative rather than a summative evaluation. This approach was followed for several reasons.
First, the Ph.D. program at UTEP has been in place for approximately five years. Given that on
average most Ph.D. candidates complete their training in 6 years it is unreasonable to expect that
the Ph.D. program will have met its objectives during its fifth year of implementation. Secondly,
the Ph.D. program is the first of its kind for the department of psychology and as a result will
undergo procedural changes commonly associated with the development of new programs in any
Program Evaluation Proposal 5
organization. Finally, this evaluation seeks to provide evaluation information that can be utilized
to improve the program rather than assess its absolute merit or worth at this point. Accordingly,
this evaluation aims to provide information that can be utilized by program administrators,
faculty, staff and students of the UTEP doctoral program in psychology to improve the program
and address issues of concerns.
Evaluation Questions
Specifically, this evaluation proposes to examine the following evaluation questions: (1)
To what extent has the program been implemented as proposed? (2) To what extent has the
program progressed toward meeting its stated objective of providing trained “bilingual/bicultural
research psychologists”? (3) What are the perceptions of the various stakeholder regarding the
Ph.D. program? and (4) What are the main concerns for program development and improvement
among the faculty and students of the program? These evaluation questions have been selected
to represent broad areas of interests that all stakeholders (e.g., local staff and faculty, students in
the program as well as higher level university and public officials) share in common. In
addition, these questions can provide valuable information that can be directly fed back to the
program as well as enable the participants of the program to voice issues of concerns and make a
worthwhile contribution toward the improvement of the graduate education offered at UTEP.
Evaluation Design
This evaluation concerns both process and implementation issues. That is, this evaluation
will assess whether the Ph.D. program has been implemented as proposed and whether the
program has made progress in that process. Shortell & Richardson (1978) have argued that
“recurrent institutional cycle designs provide a flexible way of handling some major threats to
internal validity and is particularly conducive to settings and circumstances in which new groups
Program Evaluation Proposal 6
of individuals are exposed to ongoing programs” (p. 63). The Ph.D. program at UTEP certainly
would fit this description. The UTEP program has been in place for approximately five years
and has had a new cohort of students entering the program each year since it began. This design
allows for the integration of data from multiple sources (e.g., program participants at various
stages of the programs, faculty, and staff) while allowing for diversity in information. This
design is made up of three designs including the one-shot case study, a static group comparison
design and the single group pre-posttest design. This design offers several advantages in that it
controls for the effects of history, testing , instrumentation, selection and attrition. However, it
does not rule out the effects of maturation or regression. Because this evaluation is not
concerned with generalizing outside of UTEP at this time, external validity issues (e.g.,
generalizing to other participants, settings, times) are not of major concern.
Population and Sampling Process
This evaluation will be conducted with the program participants at the University of
Texas at El Paso. The proposed evaluation will include both paper and pencil assessments of
participants satisfaction with the program, supervision, coursework, research and other areas of
the program. In addition, focus group interviews will be conducted with each student cohort
enrolled since the program was installed as well as with both the faculty and staff of the program.
Focus group interviews will be carried in one of the seminar rooms of the department. These
rooms are generally medium in size and can accommodate up to 20 people. As recommended by
focus group researchers (Krueger, 1994), the sessions will be tape-recorded to ensure the
accuracy of the data.
Participants. Participants will include students, faculty and staff of the Ph.D. program at
the University of Texas at El paso. Participant will be asked to complete both a written survey
Program Evaluation Proposal 7
and a focus group interview designed to assess their perceptions and attitudes about the Ph. D.
program at UTEP. In addition, individual interviews will be conducted with various university
officials including the Dean of the Liberal Arts College, University Vice President of Academic
affairs and the UTEP President to assess their knowledge about the programs’ success.
Individual interviews will also be conducted with other University officials that are presently
involved with the program or were involved in its formulation, or development.
Measures
Participant survey. Participant perceptions about the program will be assessed through a
survey questionnaire (see Appendix A). This survey will assess various aspects of the program
including participants knowledge, development and satisfaction (e.g., satisfaction with the
program, supervision).
Participant knowledge. Participants’ knowledge about the program will be assessed with
a scale specifically designed for this project. The scale is comprised of four items that are
presented in Likert format. Participants rate the extent to which they agree or disagree with each
item on a scale ranging from one to five. Scoring is accomplished by summing across the four
items. With a five-point response scale, scores can range from a low of 4 to a high of 20. Higher
scores on the scale indicate greater knowledge about the program.
Development. Participant development will be measured using a scale designed
specifically for this evaluation. The scale is comprised of 6 items that are presented in Likert
format. Participants rate the extent to which they agree or disagree with each item on a scale
ranging from one to five. Scoring is accomplished by summing across the 6 items. With a five-
point response scale, scores can range from a low of 6 to a high of 30. Higher scores on the scale
indicate more positive views about participant development in the program.
Program Evaluation Proposal 8
Satisfaction. Satisfaction with the program, supervision, co-workers will be measured
using a scale modelled after the Job Descriptive Index (JDI) developed by Smith, Kendall, &
Hulin (1969). Each scale contains an 18-item index that presents respondents with adjective
checklists. Respondents indicate "yes" if the adjective describes their work (or co-worker,
supervision, pay), "no" if it does not describe their work (or co-worker, supervision, pay), and
"?" if they cannot decide.
Focus groups. Focus group interviews will be administered to individual stakeholder
groups (see Appendix B). The focus group interview is composed of four parts. Part I is
designed to determine the participant's attitudes and feelings toward program. It contains three
questions dealing with the perception about the Ph.D. program (e.g., What is your general
perception of the program? What do you think of it?). Part II is designed to explore participant's
knowledge of the program. It contains several questions that assess the participant's knowledge
and experience with the program (e.g., What do you perceive as the purposes or guiding
philosophy of the program?). Part III and IV are designed to understand participant's concerns
about the program. It contains questions that deal with participant's feelings about the program,
its objectives, goals and overall purposes (e.g., What are some concerns that you have about the
program?).
In addition, the present evaluation will also collect archival data to assess the programs
progress. Archival data collected will include samples of students program plans for both
current and graduated students. These data will be used to examine the extent to which the
program has been implemented as proposed.
Procedures
Participants will be asked to attend a discussion session with the program evaluator. The
Program Evaluation Proposal 9
participants will be scheduled to attend their session with their respective cohort (e.g., 1st. year
students, faculty, staff). At the begining of the session the evaluator will give a brief
introduction and describe the purpose of the session in general terms. Participants will be
informed of their rights as participants. They will be told that their responses are anonymous and
only the evaluator will have access to their individual responses. Each participant will then be
given a questionnaire to complete and place into a box located near the entrance of the room.
When the group is finished, the evaluator will then begin the focus group discussion with the
group.
Focus group interviews will be conducted in one of the seminar rooms in the department
of psychology building by the evaluator and an assistant. Again, participants will be informed of
their rights and they will be told that their responses are to be kept anonymous. Participants will
be told not to state their name or any other identifying information during the focus group
interviews. Approximately 60-90 minutes will be required to complete the focus group
interviews. However, should more time be needed it will be determined at the time of the
interview with the consent of all of the participants. Upon completion of the interview,
participants will be given the opportunity to ask questions regarding the purpose of the
evaluation. All responses and comments about the interview will be recorded. The tape recorder
will be turned off by the evaluator when the last participant has left the room.
Analysis
Appendix C outlines the evaluation questions along with the information required for
each question, the information source and the method of data collection. As can be seen the
evaluation questions will be answered using a combination of quantitative and qualitative
information. For the quantitative data reliability indices will be calculated to determine the
Program Evaluation Proposal 10
internal consistency of the items for all the scales included in the questionnaire. Scores on each
of the scales will be used to compare across participant groups as well as to the narrative
responses from the focus group interviews.
Qualitative survey comments and focus group interview responses will be content
analyzed and major themes will be identified for the entire group (e.g., including all participants)
as well as for each group separately (e.g., student responses, faculty responses etc.). The data
will be coded and analyzed by two independent judges to cross check the reliability of the coding
procedures. Major themes and concerns will be compiled for comparisons between the
stakeholder groups.
Personnel, Equipment & Logistics
The evaluation design described in the design section, will be conducted with the
program participants at the University of Texas at El Paso. The university is located near the
Mexico-US border approximately 3-5 miles from Ciudad Juarez, Mexico. It is a midsize
university with a predominantly Hispanic student population from the surrounding communities.
Both the participant survey and focus group interviews will be carried in one of the
seminar rooms of the department (e.g., Psych. 308). These rooms are generally medium in size
and can accomodate up to 20 people. As recommended by focus group researchers (Krueger,
1994), the sessions will be tape-recorded to ensure the accuracy of the data. Participants will be
asked to attend a discussion session with the program evaluator. The participants will be
scheduled to attend their session with their respective cohort (e.g., 1st. year students, faculty,
staff).
Evaluation time line. Approximately seven months will be needed to conduct the full
evaluation. This time will be used to develop measures, conduct observations, administer the
Program Evaluation Proposal 11
survey, analyze data, review with clients and to prepare and present a final report. A significant
portion of time will dedicated to the gathering of focus group interviews (e.g., 2 months) as well
as to the analysis of this data. All other aspects of the evaluation will last no longer than one
month. Although some personnel in the department of psychology may be qualified to assist in
the evaluation, because of the nature of the data being collected and confidentiality issues the
expertise of individuals in the department will not be utilized.
Implications & Recommendations
The results from both the questionnaire survey and the focus group interview will provide
data on stakeholders assessment of the Ph. D program at UTEP. The result will help determine
whether the program has made progress toward achieving its goals. Furthermore, the findings
will also help to identify program strengths and weaknesses. Specifically, the findings of the
proposed evaluation will provide information on (1) the extent to which the program has been
implemented as proposed; (2) the extent to which the program has made progress toward
meeting its stated objectives; (3) provided information about the perceptions of stakeholders
regarding the Ph.D. program; and (4) help identify areas of concern for the purposes of program
development and improvement. Furthermore, this evaluation plan can provide information that
can guide the development of similar programs in other schools. This is particularly important
since the strength of the Hispanic community is rapidly increasing and the need for such
programs may become acute.
Human Subjects
The proposed evaluation plan aims to assess the Ph. D. program at UTEP. Participants in
this project will be required to complete an anonymous survey assessing their perceptions about
the Ph.D. program as well as a focus group interview with other participants. Participants will
Program Evaluation Proposal 12
include students, faculty and staff as well as other selected university and public officials
associated with the program. Local stakeholders (e.g., students, faculty, staff and other UTEP
personnel) will be asked to participate in a discussion with the program evaluator. A flyer with a
brief description of the project will be placed in individual mailboxes as well as throughout
various information boards in the department. The flyer will include a brief description of the
project, as well as the date, time, and location of the study.
Questionnaires will be administered to participants on the same day as the scheduled
focus group discussions. Participants will be informed of their rights as participants. They will
told that responses are anonymous and only the evaluator will see the completed questionnaires.
Instructions will be read aloud and participants will be given a pencil, an informed consent form,
the survey questionnaire. Participants will be told not to write their name or any other
identifying information on the survey itself. Once completed the participants will place their
individual surveys into a box located near the entrance of the room. After a brief break, focus
group discussion will begin.
Focus group interviews will be conducted in one of the seminar rooms in the department
of psychology building by the evaluator and an assistant. Again, participants will be informed of
their rights. Participants will be told not to state their name or any other identifying information
during the focus group interviews. Approximately 45-60 minutes will be required to complete
the focus group interviews. Upon completion of the interview, participants will be given the
opportunity to ask questions regarding the purpose of the evaluation. All responses and
comments about the interview will be recorded. The tape recorder will be turned off by the
evaluator when the last participant has left the room.
Program Evaluation Proposal 13
References
American Psychological Association. (1994). Graduate study in psychology. Washington, DC:
Author.
Jones, J. M., Keppel, G., & Meissen, G. (1993, February). Site visit report: Department of
Psychology University of Texas at El Paso. Berkeley, CA.
Krueger, R. A. (1994). Focus groups (2nd. ed.). Thousand Oaks, CA: Sage.
Marin, G., & Marin B. V. (1991). Research with Hispanic populations. Newbury Park, CA.
Sage.
Norcross, J. C., Hanych, J. M., & Terranova, R. D. (1996). Graduate study in psychology: 1992-
1993. American Psychologist, 51, 631-643.
Purdy, J.E., Reinehr, R.C., & Swartz, J.D. (1989). Graduate admissions criteria of leading
psychology departments. American Psychologist, 44, 960-961.
Ronco, S., Passmore, B., Morales, T., & Schwartz, O. (1997). The University of Texas at El
Paso fact book 1996-1997. El Paso, Texas. The University of Texas, Office of
Institutional Studies.
Shortell, S. M., & Richardson, W. C. (1978). Health program evaluation. Saint Louis, MO. The
C. V. Mosby Company.
Smith, P. C., Kendall, L. M., & Hulin, C. L. (1969). The measurement of satisfaction in work
and retirement. Chicago, IL. Rand Mc Nally.
The University of Texas at El Paso (1992). Doctor of philosophy in psychology. El Paso, Texas.
University of Texas
Program Evaluation Proposal 14
Table 1. Process Model of Evaluation.
Preexisting Conditions Program Components Intervening Events Impact/Consequences
No Ph.D. programs
exist to meet the needs
of the Hispanic
Population
Ph.D. programs in
Texas are beyond their
estimated capacity to
train additional
psychologist for the
state.
There is no Ph.D.
program available to the
El Paso-Juarez
community.
-Inputs/Obj.
-Prepare research
psychologist to serve
the English-Spanish
Bicultural
communities in the
Southwest.
Resources
-Graduate Faculty
-Adequate facilities
-Adm. Support
-Library resources
-Computing resources
-Extramural funding
Activities
-External evaluations
have been conducted.
-Recruitment of students
has continued.
Internal
-Faculty Turnover
-Budgetary changes
-Student dropout rate
-Availability of
assistantships
-Student progress
External
-Extramural funding
-Poor applicant pool
-Low visibility
Proximal
-increased pool of
qualified applicants
-increased enrollment
in the Ph.D. program
-increases in
applicants from the El
Paso/Juarez area
-increased fluency in
Spanish language
among students
-increases in the
number of students
passing the language
proficiency exam.
-increases in the
passing rate of
students taking the
first year exam
-increases in the rate
of students reaching
ABD
-increases in
extramural funding
awards to faculty
Distal
-Increase the number
of bilingual/bicultural
applied research
psychologists
-Greater cooperation
/communication
between UTEP and El
Paso-Juarez
community
-Increase KSAs of
Psychologist
Program Evaluation Proposal 15
APPENDIX A
Participant Questionnaire
Program Evaluation Proposal 16
Survey Questionnaire
We are interested in learning how to improve the Ph.D. program at UTEP. Please give your honest opinions to the questions asked here. Information collected on this survey will assist in the evaluation of the Ph.D. program and will be used to make improvements. All information is confidential and only the program evaluator will have access to the information you provide--you as an individual will be anonymous.
Directions: Read each statement carefully and check the box that best represents your opinion next to each statement.
Participant’s Knowledge Strongly Disagree
Disagree Neither Agree Nor Disagree
Agree Strongly Agree
There are clear guidelines on the program requirements. I am informed of program changes affecting my educational goals
Information is shared with students is timely and organized. There is adequate information/resources available to students.
Participant Development Strongly Disagree
Disagree Neither Agree Nor Disagree
Agree Strongly Agree
Student research is taken seriously in the department. I would like to influence decisions impacting students.
I find the coursework in the program worthwhile. I find the coursework in the program useful.
There is good cooperation among faculty and students. Recently, I have considered leaving the Ph. D. Program.
Directions: Read each statement carefully and check the box that best represents your opinion next to each statement.
PROGRAM IN GENERAL Think of the Ph.D. program at UTEP. How well do the words or phrases describe the program in general?
Yes No Don’t Know
CURRENT POSITION Think of the work you do at present in the Ph.D. program. How well do the words or phrases describe what your position is like most of the time?
Yes No Don’t Know
Pleasant Fascinating Bad Routine
Ideal Satisfying Waste of time Boring
Good Good Undesirable Creative
Worthwhile Respected Worse than most Uncomfortable
Acceptable Pleasant Superior Useful
Better than most Tiring Disagreeable Healthful
Makes me content Challenging Inadequate Too much to do
Excellent Frustrating Rotten Simple
Enjoyable Repetitive Poor Gives sense of accomplishment
Program Evaluation Proposal 17
Directions: Read each statement carefully and check the box that best represents your opinion next to each statement.
SUPERVISION
Think of the Ph.D. program. How well do the words or phrases describe the supervision you receive in the program?
Yes No Don’t Know
COWORKERS-PEERS
Think of the Ph.D. program. How well do the words or phrases describe your coworkers or peers?
Yes No Don’t Know
Asks my advice Stimulating Hard to please Boring
Impolite Slow Praises good work Helpful
Tactful Stupid Influential Responsible
Up-to-date Fast Doesn’t supervise enough Intelligent
Has favorites Easy to make enemies Tells me where I stand Talk too much
Annoying Smart Stubborn Lazy
Knows job well Unpleasant Bad Gossipy
Intelligent Active Poor planner Narrow interests
Around when needed Loyal Lazy Stubborn
Please use this space to give us your reactions and views of the Ph.D. program at UTEP.
1. Do you like the content of the program? Yes No
Why or why not? _________________________________________________________________________________________________
_________________________________________________________________________________________________
_________________________________________________________________________________________________
2.. Did you like the faculty of the program? Yes No
Why or why not? _________________________________________________________________________________________________
_________________________________________________________________________________________________
_________________________________________________________________________________________________
3. What did you like most about the training you have received thus far?
_________________________________________________________________________________________________
_________________________________________________________________________________________________
______________________________________________________________________________________________ ___
4. What do you like least about the training you have received thus far? _________________________________________________________________________________________________________________________
_________________________________________________________________________________________________
_________________________________________________________________________________________________
Program Evaluation Proposal 18 5. Are there any areas covered in the program that you feel should have been covered in more detail?
_________________________________________________________________________________________________
_________________________________________________________________________________________________
_________________________________________________________________________________________________
6. Would you like to see other courses added to the program? Yes No If YES, in what area? _________________________________________________________________________________________________
_________________________________________________________________________________________________
_________________________________________________________________________________________________
7. Of the courses you have taken so far, indicate those which have been the most useful: _________________________________________________________________________________________________
_________________________________________________________________________________________________
_________________________________________________________________________________________________
8. Of the courses you have taken so far, indicate those which have been the least useful: _________________________________________________________________________________________________
_________________________________________________________________________________________________
_________________________________________________________________________________________________
THIS INFORMATION IS CONFIDENTIAL AND WILL ONLY BE USED TO ANALYZE THE RESULTS. IT WILL HELP US MAKE IMPROVEMENTS ON THE TRAINING. DEMOGRAPHICS
Gender: Male Female
Student Status:
Ph. D. Student
Health Track Human Behavior in Organization Track
MA Student
General Experimental Clinical
Number of years in the program:
Ph. D. Student
1st. year. 2nd. year. 3rd. year. 4th. year. 5th. year. 6th. year.
MA Student
1st. year. 2nd. year. 3rd. year. 4th. year. 5th. year. 6th. year.
Program Evaluation Proposal 19
APPENDIX B
Focus group interviews
All of you have been asked to participate in this study because you can help me understand how
people like yourselves [e.g., students, faculty, staff, or other] feel about the Ph.D. program
offered by the department of psychology. We are interested in finding out what you feel about
the Ph.D. program--be that positive or negative. There are no wrong or right answers; we just
want to know what you think and feel about the program. Remember your answers are
completely private. No one except the evaluator (myself) will have access what you say today.
To begin with we are going to start by completing a brief survey that explores how you feel
about the program. We will begin our discussion as soon as that is completed. Take a few
moments and answer the survey then place it face down on your table so that I know when it is
alright to begin.[WAIT FOR PARTICIPANTS TO COMPLETE THE SURVEY].
Now I am going to turn on the tape recorder and check to make sure it is functioning properly. I
would like all of us to count to five to check the tape recorder, I will begin and the next person to
my right continue till we have gone full circle. I will then check to make sure the recorder is
functioning properly and begin the discussion. To keep your answers private, we will not use
your name or any other type of identifying label. [REWIND TAPE AND PLAY BACK TO
ENSURE THAT THE RECORDINGS ARE AUDIBLE].
Part I. Goal: Determine attitudes and feelings toward the Ph.D. program.
1. What is your general perception of the program?
--What do you think of it? Do you think well of it?
2. What do you like about it? What do you think the program does well?
--What makes you say that?
3. What do you think is bad about the program? What don’t you like about the program?
--What makes you say that? Explain ?
Part II. Goal: Explore participants knowledge of and about the program.
4. What do you perceive as the purposes (goals, objectives) or guiding philosophy of the
program?
--How did you arrive at this answer ?
--What kinds of things happened that told you this?
5. What do you think about this philosophy? Do you agree with these purposes or philosophy?
Do you think the problems the program addresses are severe? important?
Program Evaluation Proposal 20
6. I am going to read to you the stated objectives of the program as written in the “gray book”--
the program’s official plan that delineates the purpose and objectives.
Read [The Doctor of Philosophy (Ph.D.) program at UTEP is designed to
produce bilingual/bicultural research psychologists that will be able to serve
Hispanic populations in Texas. The Ph.D. program at UTEP aims “to prepare
research psychologists to address questions applicable to the English-Spanish
bicultural communities in the Southwest” (The University of Texas at El Paso,
1992, p. 8)].
7. What do you think about this philosophy? Do you agree with these purposes or philosophy?
Do you think the problems the program addresses are severe? important? Is it different from
what you expected? Why or why not?
8. What do you think the theory or model for the program is? Why/how do you think it works?
How is it supposed to work? Why would the program actions lead to sucess on the program’s
objectives or criteria? Which program components are most critical to success?
9. Typically, what types of activities do students in the department engage in? (academic
activities)?
10. Are these activities different from that of other students in other schools? Explore ?
11. What does a student have to do to finish the program?
Part III. Goal: Understand participant concerns about the Ph.D. program
13. What concerns do you have about the program? About its outcomes? Its operations? Other
issues?
--What makes you say that?
14 If there any changes that you would like to see occur what would they be?
How would these help you or others in the program?
Do you think others share your concerns?
Do you think these would be easily implemented?
15. Why do you think these changes have not occurred ?
Program Evaluation Proposal 21
Part IV. Goal: Learn about issues related to specific participant groups
Faculty
A. Is the program being implemented as originally conceptualized?
B. Are there any problems with the implementation process?
C. What problems do faculty perceive?
D. How satisfied are they with the way the program is implemented”
E. How satisfied are they with the quality of students being enrolled?
F. How satisfied are they with the graduating students?
G. How well does the program prepare its students for the job market?
H. How comparable is the curriculum to similar graduate programs (nationally, or regionally)?
Students
A. Is the program meeting their expectations? Why or why not?
B. What are some problem areas in the program:
1. Classes: Have classes been offered regularly? Which have and which have not? are there
any additional courses that you would like to see offered?
2. Internships? How many students have gone on internships? How satisfied were they with
their experiences? How relevant were the internships to their career/professional goals?
3. Funding? How are funding decisions made?
4. Teaching opportunities? How many have taught classes? How is the assignment of
courses made? What preparation do students receive to teach a course?
5. Other?
Staff
A. Has there been added work as a result of the Ph.D. program beyond what is possible with the
resources available to the department?
General
A. Has the program graduated the first class of students? How prepared are they for the market?
How well does the program prepare its students for the job-market?
B. How comparable is the curriculum to those of other graduate programs (nationally, or
regionally)?
C. Has the program met accreditation standards (p. 29)?
16. Are there any comments or suggestions that you want to add ?
Program Evaluation Proposal 22
APPENDIX C
Evaluation Plan Outline
Evaluation
Question
Information
Required
Source of
information
Method
1. To what extent
has the program
been implemented as
proposed?
Qualitative Participant interviews
Archival data
Focus group interviews
Departmental records
2. To what extent
has the program
progressed toward
meeting its stated
objectives?
Qualitative Participant interviews
Archival data
Focus group interviews
Departmental records
3. What are the
perceptions of
stakeholders
regarding the
program?
Quantitative/
Qualitative
Participant survey
Participant interviews
Participant survey
Focus group interviews
4. What are the
main concerns for
program
development and
imrovement?
Quantitative/
Qualitative
Quantitative/
Qualitative
Participant survey
Focus group interviews