Summer end 1

Project 1 Program Evaluation Critique: Project 1involves reviewing and critiquing the program evaluation study by James-Burdumy and colleagues (see below). The project is worth 15% of the total grade and requires that you provide an overview and critique of the evaluation by James-Burdumy et al (2016) that includes critiquing (a) the background and focus of the evaluation; (b) the evaluation methodology; (c) the analytical approach and findings; and (d) conclusions and recommendations.

James-Burdumy, S., Beyler, N., Borradaile, K., Bleeker, M, Maccarone, A., & Fortson, J. (2016). The impact of playworks on students' physical activity by race/ethnicity: Findings from a randomized controlled trial. Journal of Physical Activity & Health, 13 (3), 275-280.

Deliverable : A written critique of James-Burdumy et al (2016) addressing elements below. Critiques should be type-written, doubled-spaced in 12pt font.

(a) Background and Focus. What is the background and context for the evaluation? What program is being evaluated? Describe its components of the logic, model or theory of the program (e.g., resources, activities, outputs, outcomes, impact; Mertens & Wilson, 2019, pp. 229-242)? What were the goals, objectives, and purposes of the evaluation? What question(s) were being answered in this evaluation? See Mertens and Wilson (2019) Chapter 7-8.

(b) Evaluation Method. Data—describe the data used to assess the evaluation objectives (e.g., what types of data are collected? how much data is collected and why?) and comment on the appropriateness of these data (e.g., What were the constructs of interest and how were these measured? Is there evidence of the reliability and validity of the measured constructs to support their use? Are these appropriate given the evaluation goals, objectives and purposes?) See Mertens and Wilson (2019) Chapter 10. Evaluation Design—Identify and describe the evaluation design (e.g., Mertens & Wilson, 2019, Chapter 9). Explicate how the evaluation design relates to the goals, objectives, and purposes of the evaluation (see Mertens & Wilson, 2019, Chapter 8) and comment on the appropriateness of the design to answer the evaluation objectives (e.g., Is the design appropriate to the evaluation goals, objectives and purposes? Why do you suppose they chose this design over other alternatives? Which sources of potential invalidity have they ruled out with this design and what sources might remain? What might be some important considerations other than the pros and cons of the designs themselves? Would you have used a different design? Why or why not?). See Mertens & Wilson, 2019, Chapters 8-10.

(c) Analyses and Findings. Analyses—Describe the analytical approach taken to evaluate the data and comment on the appropriateness of the analyses in relation to the evaluation goals, objectives and purposes (e.g. What analyses were used and how did these address the evaluation goals, objectives and purposes? Are there other types of analyses that might further inform the evaluation goals, objectives and purposes?). Findings— Describe the major finding of the evaluation and comment on how the findings relate to the evaluation goals, objectives and purposes (e.g., What are the main findings of the study? How do the findings address the evaluation goals, objectives and purposes? Do the findings support, refute or inform the program? What alternative interpretations of the findings can be ruled out and what plausible alternative or rival explanations can be made? What limitations or qualifiers must be placed in the evaluation results given conceptual/theoretical, methodological or statistical concerns?). See Mertens & Wilson (2019) Chapter 12.

(d) Conclusions and Recommendations. Describe the major conclusions and recommendations of the evaluation and comment on the evidence supporting, refuting or informing the evaluation goals, objectives and purposes (e.g., What were the major conclusion and recommendations? How do the conclusion and recommendations address the evaluation goals, objectives and purpose? Are there any conceptual/theoretical, methodological/statistical or practical concerns warrant caution?). See Mertens & Wilson (2019) Ch.12.

4

Project 1 Evaluation Form

Individual Project 1 Program Evaluation Critique. Individual Project 1 required you to provide an overview and critique of the evaluation study conducted by James-Burdumy et al (2016). The assignment should review and critique (a) the background and focus of the evaluation (3 pts); (b) the evaluation method (3pts); (c) the analytical approach and findings (3pts); (d) conclusions and recommendations (3pts) using questions described in Appendix A; in narrative form within no more than 3-5 single-spaced pages; and (e) be written in a clear and coherent manner that critiques all elements of the program evaluation (3pts). The assignment is worth 15% of the total grade.

Background & Focus (3 Points)	Not at all 0	Partially .5	Completely 1	Total Points
a. Does the narrative describe the background and context for the evaluation?
b. Does the narrative describe relevant aspects of the program being evaluated? (e.g., its components, logic model or program theory)
c. Does the narrative describe the purpose of the evaluation? (e.g, goals, objectives, and evaluation questions)
Evaluation Method (3 Points)	Not at all 0	Partially .5	Completely 1	Total Points
d. Does the narrative describe the data used to assess the evaluation objectives? (e.g., types and amount of data collected and why?)
e. Does the narrative comment on the appropriateness of the data? (e.g., relevance of data to goals, objectives and purposes, operationalization of constructs, measurement qualities--reliability and validity)
f. Does the narrative describe the evaluation design and comment its appropriateness for the evaluation (e.g., appropriateness of the design given the goals, objectives and purposes; advantages and disadvantages of the design— threats to validity, alternative designs, pros and cons)
Analysis and Findings (3 Points)	Not at all 0	Partially .5	Completely 1	Total Points
g. Does the narrative describe the analytical approach taken to evaluate the data and comment on the appropriateness of the analyses (e.g. analyses used to address goals, objectives and purposes; alternative analyses that might further inform the evaluation goals, objectives and purposes?)
h. Does the narrative describe major findings of the evaluation in relation to the evaluation goals, objectives and purposes (e.g., What are the main findings of the study? How do the findings address the evaluation goals, objectives and purposes?)
i. Does the narrative comment on how the findings relate to the evaluation goals, objectives and purposes (e.g., Do the findings support, refute or inform the program? Alternative interpretation of findings? Any limitations or qualifiers to the findings?)
Conclusions & Recommendations (3 Points)	Not at all 0	Partially .5	Completely 1	Total Points
j. Does the narrative describe and discuss the major conclusions? (e.g., What were the major conclusion and how do they address the goals, objectives and purposes of the evaluation?)
k. Does the narrative describe and discuss recommendations? (e.g., What were the recommendations and how do the recommendations address the evaluation goals, objectives and purpose?)
l. Does the narrative comment on the evidence supporting, or refuting the program goals, objectives and purposes (e.g., Are there any conceptual/theoretical, methodological/statistical or practical concerns warrant caution?)
Critique (3 Points)	Not at all 0	Partially .5	Completely 1	Total Points
m. Does the narrative critique of the background, context or purpose?
n. Does the narrative critique of the methodology or analytical approach?
o. Does the narrative critique the results, major findings or conclusions?
Grand Total