''FOR SMART GURU ONLY''

3. In the random assignment evaluation of the New York City voucher program, only 75% of students who were offered a voucher actually used it to attend private school. Suppose you are interested in evaluating the impact of using the voucher to attend private school, not just the impact of being offered the voucher.

(a) One option would be to measure outcomes for only those members of the treatment group who used the vouchers to attend private school and then compare those to the outcomes for the control group, which was not offered the voucher. Would this yield an internally valid estimate of the impact of the voucher use? Why or why not?

It will not be effective method in my opinion, as it will only depend only one dependent variable to carry the investigation. There will hidden variables that will not be considered in the experiment and the data will be considered a bias and the outcome will not be effectively generalized.

(b) Describe a different way in which you may try to generate an internally valid estimate of the impact of using the voucher?

Section II. Longer Answer ( 2-3 paragraphs each )

1. Imagine that tomorrow’s Providence Journal contains an article titled “Lower Class Sizes Reduce the Achievement Gap”. The article reports on results of a secret randomized experiment in Rhode Island that involved randomly assigning students to large or small classes. Being in a small class improves student achievement, narrowing the achievement gap in these schools by 10%. Two experts in the article take different views on the findings – one argues that the initiative is a failure because it does not close the achievement gap; another argues that the initiative is a success because it demonstrated important improvement. Given your knowledge from the class, you are going to write an editorial arguing that these experts are missing the real issue, whether the benefits of the program outweighed the costs. However, you know that any cost-benefit analysis relies on getting a good estimate of the impact of the program. You want to read the report carefully to understand better how the randomized experiment was conducted.

Identify three important sources of bias that you might be looking for as you read the report. What threats to internal validity might you be concerned about in these estimates? Describe the threat, provide a concrete example of each, and explain how this threat to validity might affect the estimates (would it bias them upwards, downwards, or does it depend). Be sure to justify your answers carefully for each of these three threats. [Hint: This question is about the threats to validity in a randomized experiment, not about cost-benefit analysis].

2. Imagine that you are a highly-paid consultant hired by Brown University to conduct an impact evaluation of the UEP program. In particular, the University wants to understand whether participating in the program increases the earnings of program graduates and, if so, by how much. The results will be quite helpful in future recruitment efforts. You were hired for your intimate knowledge of the program and for your exceptional skills in program evaluation.

The University asks you to present an evaluation plan, laying out your two most promising strategies for evaluating this program. Obviously, there will be trade-offs involved in each approach. Describe briefly two realistic strategies that you might use to evaluate the program, and discuss (briefly) the major strengths and weaknesses (relative to the alternatives) of each approach.

Section III. Case Example ( one brief paragraph each )

In 1965, the federal government provided funding for Head Start to the 300 poorest counties in the U.S. In order to understand whether this funding was effective, Ludwig & Miller examined whether this additional funding increased Head Start participation. Essentially, they used a regression-discontinuity design to compare Head Start participation rates in counties above and below this cutoff (the cutoff was at a county poverty rate of 59.198% -- counties that had higher poverty rates than this got the additional funding).

In Table A on the following page, I reproduce some of their results. Using this table and your knowledge of regression-discontinuity designs, answer the following questions [NOTE: you do not need to read Ludwig & Miller’s article]:

1. Describe in words the specific comparison that Ludwig & Miller are making (i.e., what will they compare to estimate their causal effect). Write out a regression model that they could use to estimate this effect. Be sure to define your variables.

2. Using the results from Table A, how do you interpret the parameter estimate in the 3rd row,

3rd column (0.172)?

3. The authors present results using a range of different bandwidths. Why do they do this?

Based on these results, would you say that receiving the additional funding increased Head

Start participation and spending, or not? Support your answer.

BONUS QUESTION

4. Why do you think the authors looked at the effect on non-Head Start spending (the last row),

something that should not have been affected by additional Head Start funding? What might this result tell us?

Table A (adapted from Ludwig & Miller, 2007): Effect of Head Start Assistance on Head Start Spending and Participation.

Each cell contains an estimate of the estimated effect from the RDD, a standard error (in parentheses), and a one-sided p-value [in brackets].

Variable Control Group

Mean

RDD Estimates by Different Window Widths