evaluation and methodology needed for grant

Monitoring & Evaluation Plan

Design. The evaluator will conduct a context, input, process, product (CIPP) evaluation

(Stufflebeam, 2007). The CIPP model provides long-term evaluation of program design to

determine how programmatic features meet participant needs within the contexts within which

participants live and learn. Additionally, the CIPP model studies the program implementation

process to provide ongoing formative feedback for program administrators and staff to utilize in

programmatic revisions and improvements. Finally, the CIPP model assesses program outcomes

through the analysis of data to determine overall programmatic impact. The evaluator will utilize

the CIPP model in a cyclical manner, revisiting all four parts of the model throughout the grant.

This approach will provide continual monitoring of the overall grant implementation process and

measure the achievement of outcomes, and overall program goals.

Participant Consent. The evaluator and PI will collaborate to acquire written consent for all

participants in the evaluation. Consent forms will be included in a program packet, and will be

part of program registration. Participation is voluntary, and no program participant will be

excluded from the program if they choose not to participate in the evaluation. The Institutional

Review Board (IRB) at Ball State University will review and monitor the evaluation. The

evaluator and PI have completed training on ethical standards for maintaining confidentiality and

protecting human subjects through Ball States’ IRB.

Guiding Questions. The evaluation will be guided by the following research questions:

1. Context: To what extent is the program targeted to meet important Afghan junior faculty needs?

2. Input: To what extent is the program’s structure including curriculum, instruction, and policies aligned with the program as proposed in the grant?

3. Process: To what extent is program implementation consistent with the program as proposed in the grant?

4. Product (Outcomes and Goals): To what extent is the program effective in achieving progress toward outcomes and goals as described in the Grantee Monitoring and Evaluation

Tool and detailed in the grant proposal?

Performance Indicators. Quantitative and qualitative methods will be utilized to collect data to

answer the guiding questions. Quantitative methods will include:

• The participant survey will ask junior faculty to report their perceptions regarding key outcome areas for the program. This survey will be administered in an online format using

Qualtrics at the beginning, middle and end the end of the program to provide both baseline and

progress data toward achieving the outcomes for the program. The online format is expected to

improve the response rates since the survey will be easy to access and easy to complete, even

from a mobile phone. In addition, multiple reminders may be emailed to participants with

relative ease. Participants may also complete the survey during program sessions.

• Participant data will be collected to measure progress toward outcomes as defined within the Grantee Monitoring and Evaluation Tool. Data will include participant demographic

information and program attendance.

Qualitative methods will include focus group interviews, program observations, and document

review:

• Focus group interviews will be conducted at the end of the third week of the program with participants to explore their needs, as well as perceptions of the program.

• Observations of daily program activities will be carried out during the first three weeks of programming to illustrate program implementation. Additionally, observations will provide

data used to assess alignment with best practices in faculty development.

• Programmatic documents including agendas, curricula, communications, participant work samples, etc. will be collected and reviewed throughout the grant to evaluate alignment of

program implementation to the proposed program design, and to assess achievement of

program outcomes.

This table indicates specific data collection methods utilized to answer the research questions.

Questions Performance Indicators

To what extent is the

program targeted to

meet important

Afghan junior faculty

needs?

-Programmatic policy and design documents will be analyzed for

their achievement of program outputs.

Focus group interviews with participants will be conducted to explore

the extent to which the program meets junior faculty needs through

program outputs.

-The participant survey will be utilized to assess junior faculty

perceptions regarding program outcomes.

To what extent is the

program’s structure

including curriculum,

instruction, and

policies aligned with

the program as

proposed in the grant?

-Programmatic policy and design documents will be analyzed for

their adherence to the program proposal.

-Observations of program activities will be conducted to determine

programmatic alignment with the program proposal.

To what extent is

program

implementation

consistent with the

program as proposed

in the grant?

-Observations of program activities will be conducted to paint a

picture of overall program implementation including activities and

outputs.

-The participant survey will be administered to provide program

feedback regarding program outputs.

-Program documents that describe curricula and activities, as well as

samples of participant work will be examined to determine how

program activities and outputs are realized on a daily basis.

To what extent is the

program effective in

achieving progress

toward outcomes and

goals as described in

the Grantee

Monitoring and

Evaluation Tool and

detailed in the grant

proposal?

-Participant-level data will be collected, as described in the Grantee

Monitoring and Evaluation Tool to determine how effective the

program is in achieving progress toward outcomes and goals.

-The participant survey will be administered to provide program

feedback regarding program outcomes and goals.

-Program documents including samples of participant work will be

examined to determine how the program has achieved outcomes and

goals.

-Focus group interviews will be conducted with program staff to

explore participants’ experiences related to program outcomes and

goals.

This table is in alignment with the logic model in the proposal narrative and indicates ways the

performance indicators will be utilized to monitor program goals:

Goal 1: Enhance the educational and administrative capacity of Afghan faculty.

Outputs Outcomes Performance Indicators

-Project staff complete a

written summary of the needs

assessment and SWOT

analysis

-Project staff develop an

individualized program plan

for each participant

Twelve participants begin the

program, each with an

individualized program plan

and mentor assigned

All participants participate

fully 100% of the time

-Programmatic document

review

-Program observations

-Participant data

-Participants develop syllabi

for courses taught that are

aligned with quality

assurance measures and

international standards

At least 90% of participants

report increased confidence and

skill in teaching and knowledge

of curriculum development

-Participant survey

-Focus group interview

-Programmatic document

review

-Participants develop an

action plan that includes a

wide range of new skills to

implement upon their return

home

At least 90% of participants

report increased confidence and

skill in teaching and

knowledgeable of conflict

resolution, student-focused

teaching, developing

meaningful course curricula,

social justice, equity, gender

parity/inclusion, and diversity

-Participant survey

-Focus group interview

-Programmatic document

review

Participants develop action

plans to implement changes

upon their return to

Afghanistan

At least 90% of participants

report increased knowledge of

U.S.-based best practices,

accreditation, and quality

assurance

-Participant survey

-Focus group interview

-Programmatic document

review

Goal 2: Develop long-term meaningful relationships.

Outputs Outcomes Performance

Indicators

Mentors provide meaningful

support and feedback to

participants

-At least 90% of participants

report feeling comfortable

contacting their mentor for

support

-At least 90% of participants

report their experience was

enhanced and/or more

sustainable as a result of

receiving ongoing mentorship

-Participant survey

-Focus group interview

-Programmatic document

review

Participants develop action

plans to implement changes

upon their return to

Afghanistan

-At least 90% of participants

report their experience was

enhanced and/or more

sustainable as a result of

discussions

-Participant survey

-Focus group interview

-Programmatic document

review

Participants develop plans to

implement student-centered

approaches to teaching when

they return to their home

universities

-At least 90% of participants

report enhanced knowledge of

their academic discipline and a

better understanding of

international quality assurance

standards through engagement

with U.S. and Indian faculty

and classes

-Participant survey

-Focus group interview

-Programmatic document

review

Goal 3: Enhance English language proficiency along with the pedagogic knowledge of

program participants.

Outputs Outcomes Performance Indicators

Participants write syllabi

and various plans in

English

At least 90% of participants

will demonstrate enhanced

English proficiency

-Participant survey

-Focus group interview

-Programmatic document

review

Analysis Procedures. During observations, the evaluator will take extensive field notes. Focus

group interviews will be audio recorded and transcribed in preparation for analysis. Field notes,

transcripts, and program documents will be analyzed using inductive analysis procedures (Hatch,

2002). Such analysis follows a systematic process beginning with the close examination of key

segments of the data leading to the identification of “domains that reflect relationships

represented in the data” (Hatch, 2002, p. 164). These domains are then assigned a name, or code.

Next, the codes are applied across the data including field notes, transcripts, and documents.

Finally, data ascribed to each code is summarized. Quantitative data from survey responses will

be entered and analyzed in SPSS 14.0. Descriptive statistics will be calculated for each item and

subscale. Participant-level data will be analyzed to measure program outcomes as detailed in the

Grantee Monitoring and Evaluation Tool. Descriptive statistics will be calculated using SPSS

14.0 to determine to what extent the program meets the projected performance measures.

Deliverables. Formative reports will be provided to the PI and project manager at the conclusion

of the planning period, midway through the implementation period, and at the conclusion of the

program. These reports will focus on program planning and implementation, described through

observation, focus group interview, and document data collection and analysis, as well as

participant survey data collection and analysis.

A summative evaluation report will be provided to the Public Affairs Section at the conclusion of

the grant period. This report will include, as specified in the RFP, the following: (1) evidence of

program quality; and (2) progress towards each of the stated outcomes included in the Grantee

Monitoring and Evaluation Tool. The summative report will be completed and submitted in

September 2019. This report will include a summation of findings from program observations,

focus group interviews, program documents, participant surveys, and participant-level data. The

summative evaluation report will also include per the RFP, data disaggregated to break down by

subgroup.

References

Hatch, J. A. (2002). Doing Qualitative Research in Education Settings. Albany: State University

of New York Press.

Stufflebeam, D. L. (2007). CIPP Evaluation Model Checklist. Retrieved from

http://www.wmich.edu/sites/default/files/attachments/u350/2014/cippchecklist_mar07.pd

f