Assignment

Assignment 1

Assignment Background

This assignment is based on a 2016 paper called™The Behavioralist Goes to School: Leverag-

ing Behavioral Economics to Improve Educational Performanceº by Steven D. Levitt, John

A. List, Susanne Neckermann, and Sally Sadoff. The abstract of the paper is here:

We explore the power of behavioral economics to inØuence the level of e ffort

exerted by students in a low stakes testing environment. We Ænd a substantial

impact on test scores from incentives when the rewards are delivered immediately.

There is suggestive evidence that rewards framed as losses outperform those

1

framed as gains. Nonfinancial incentives can be considerably more cost-effective

than financial incentives for younger students, but are less effective with older

students. All motivating power of incentives vanishes when rewards are handed

out with a delay. Our results suggest that the current set of incentives may lead

to underinvestment.

In this assignment, we will do a simple replication of some of their results. The basic

idea of the paper is to investigate whether providing students with the “right” incentives

to do well in school causes better performance. The “right” incentives in this context are

those informed by theories from behavioral economics, which uses insights from psychology,

sociology, and neuroscience to refine standard economic models to account for human beings

not always acting rationally. As the authors’ note,

“One of the biggest puzzles in education is why investment among many students

is so low given the high returns. One explanation is that the current set of

long-run returns does not sufficiently motivate some students to invest effort in

school. If underinvestment is a problem, then there is a role for public policy in

stimulating investment.”

In order to learn about what better policies might look like, the authors deploy three ideas

from behavioral economics: loss aversion, nonmonetary rewards, and hyperbolic discounting.

While these are all interesting,1 we are going to focus on replicating some results from the

authors’ experiment for testing hyperbolic discounting, which essentially means that people

sometimes discount the future too much. When weighing the costs and benefits of taking

an action that has immediate costs but future benefits (i.e., studying!), people sometimes

choose not to act because they have put too little weight on the (ever-so-distant) benefit.

To test whether they could get students to overcome hyperbolic discounting, the authors

showed up on the day of the test (right before the test) and randomly offered some students a

financial reward if they could improve their test score relative to last year’s score. Therefore,

if the students tried hard, they would realize the benefit immediately after the test rather

than in the distant future. Specifically, the authors visited a high school in Bloom Township

(Bloom), a small school district south of Chicago, and conducted a random experiment in

which they randomly sorted students into three groups: high incentives (the student receives

1Loss aversion implies that human beings are generally hurt more by losses than they are happy about gains, even for equivalent amounts. (The pain you feel when you lose $5 is greater than the joy you feel when you earn $5.) The idea behind nonmonetary rewards is that human beings are often motivated by something other than money, and finding that “something” can lead people to try harder at a given task. If you’re interested beyond that, look at Levitt, List, Neckermann, and Sadoff (2016). It’s posted on Moodle.

2

$20 if they improve), low incentives (the student receives $10 if they improve), and control

group (the student receives nothing if they improve).

For this assignment, we answer a few questions about this context and conduct some

simple analyses with the authors’ data.

3

Questions

1. Prior to getting to the authors’ experiment, suppose we were interested in providing a

credible answer to the following general question: “Does providing students with financial

incentives to do well on tests improve test performance?” A friend of yours proceeds to

answer this question by going to a local high school and collecting performance data on

student tests. Your friend then surveys the parents of all students in her sample, asking

parents, “On the day of the last test your child wrote, did you promise to pay him or

her at least $10 if they improved their performance relative to the last time they took a

similar test?”

Based on the answers to her survey, your friend creates the following grouping variable

for your sample of i = 1, . . . , N students:

Di =

! 1 if parents promised to pay

0 if parents did not promise to pay.

She then specifies the following econometric model to describe the test score performance

of student i:

Yi = β0 + β1Di + Ui

(a) Provide an example of a factor that could be in Ui.

(b) Explain (in words) why it is problematic to simply compare conditional means as

way of answering the research question. That is, why is it not a good idea to

compare the mean test score of students with Di = 1 to the mean test score of

students with Di = 0?

(c) Use the econometric model above to show (mathematically) how a comparison of

conditional means is biased by selection.

2. Now we move on to the authors’ experimental data. Open up the all BGS data v13.dta

Stata data file and the associated do file titled ProblemSet1 Dofile.do. (Both can be

found on Moodle.) As with any random experiment, we first want to make sure random-

ization was successful. To that end, we are going to try to confirm a result from Table

3 in Levitt, List, Neckermann, and Sadoff (2016).2

(a) Find the proportion of students who are eligible for a free lunch in the control group,

F L C , and proportion who are eligible in the low incentive group, F L

L .

2You do not have to refer to the paper for this assignment. I provide the references for those who are curious.

4

(b) Find the variance of the free lunch variable among a pooled sample of students in

the control group and students in the low incentive group (do not include students

in the other groups), σ2F L.

(c) Find the difference between the proportion of students who are eligible for a free

lunch in the control group and proportion who are eligible in the low incentive

group, F L L − F L

C .

(d) Find the variance of the estimator F L L − F LC , using the formula we discussed in

class: Var " F L

L − F L

C #

= σ2F L( 1

NL + 1

NC ).

(e) What is the value for the t-statistic for the hypothesis test that F L L − F L

C = 0?

Conduct a hypothesis test that the proportion of students who are eligible for a free

lunch in the control group is the same as the proportion who are eligible in the low

incentive group. (You can use Stata’s built-in hypothesis testing command that we

showed in class.) What do you conclude?

(f) Based on your analysis with the free lunch variable, was randomization successful?

3. Using the same data file and do file, let us now verify that offering immediate incentives

indeed leads to greater student effort, as captured by higher test scores. To that end,

we are going to try to confirm the results in columns (3) and (4) of Table 6 of Levitt,

List, Neckermann, and Sadoff (2016).3

(a) Test the hypothesis that the average test score of students in the control group and

the low incentives ($10) treatment group are the same. What do you conclude?

(b) Test the hypothesis that the average test score of students in the control group and

the high incentives ($20) treatment group are the same. (For this question, you’ll

see that in the do file I tell you to group students who received high incentives

and high incentives framed as a loss into one group, and to just call this the “high

incentive” group. ) What do you conclude?

(c) Now use only the sample of students who were either in the control group or the

high incentives treatment group (again, both students who received high incentives

and high incentives framed as a loss) and run a regression in which you regress

the test score of student i on the high incentives indicator variable. How does the

estimated coefficient on this variable relate to the difference in average test score

means from part (b)? How does the standard error of this estimated coefficient

3The authors structured their experiment such that they needed to do more sophisticated econometrics than we have done so far in order to produce Table 6. We therefore will not be able to replicate their estimates exactly, but we should be able to find the same qualitative patterns.

5

relate to the standard error of the difference in average test score means from part

(b)?

6