Consider a population of households who choose a consumption bundle q1, . . . , qJ to maximize preferences, represented by a utility function of the form

5 Nonsampling Error

Nonsampling error is a catch-all term for the variety of errors, other than sampling uncertainty, that may arise during the course of all survey-related activities.

Unlike sampling error, which reflects sample-to-sample variability and is only present in sample surveys, nonsampling error can also be present in censuses and administrative data.

You should worry about nonsampling error because it may cause systematic distortion or bias in estimation of population parameters of interest. Further, unlike sampling error, we cannot generally reduce this bias by gathering more data.

Nonsampling error may arise because of:

• sample selection, which occurs when the sampling design systemati- cally oversamples or undersamples a fraction of the population; an ex- treme form of sample selection is noncoverage, which occurs when the sampling design systematically excludes a fraction of the population;

• nonresponse, which results from failure to obtain answers to some or all survey questions (respectively, item and unit nonresponse);

• measurement error, which occurs when data are incorrectly requested, provided, received or recorded (because of poor questionnaire design, interview bias, respondent errors, or problems with the survey process), or when they are incorrectly coded, edited or imputed.

Sample selection (and noncoverage) may arise either because of design de- cisions by the survey statistician or the investigator, or because of self- selection by the individual units under investigation;

266

5.1 Noncoverage

Noncoverage may be caused by defects in the sampling frame, such as in- accuracy, incompleteness, duplications, inadequacy or obsolescence, or by field procedures (e.g., while a survey is conducted, the interviewer systematically misses certain types of households or persons). This is especially a problem for rare or elusive populations, such as homeless people.

A classic example of defective sampling frame occurs in telephone surveys, as individuals and families without phones (and, increasingly, those with cellular phones but no landline) do not enter the sampling frame.

Online web surveys may also su↵er from noncoverage problems, as internet usage depends largely on factors such as access and age.

When the available sampling frame is defective, the researcher faces two al- ternatives:

• redefine the target population to better fit the frame; this may be subject to criticism because the frame population no longer coincides with the target population;

• admit the possibility of noncoverage bias in statistics describing the target population; this bias arises because the population parameters of interest are “confounded” with parameters that determine the coverage probability.

267

5.1.1 Truncated and censored samples

Truncated and censored samples are forms of noncoverage arising when the sample design systematically excludes a fraction of the population in a way that depends on the value of some variable of interest Z.

Let D be a binary indicator of coverage, equal to one if Z assumes values in some region B and equal to zero otherwise. The fraction of the population subject to sampling is P[D = 1] = P[Z 2 B]. I say that a sample is censored if P[D = 1] is known or may accurately be estimated from the available data, otherwise I say that the sample is truncated.

In the case of a censored sample, I further distinguish between fixed and random censoring depending on whether the set of points B for which D = 0 is fixed or is itself random, e.g., depends on the value of some other random variable W .

To illustrate these concepts, I now consider two examples: top-coding (Sec- tion 5.1.2) and self-selection into the labor force (Section 5.1.3).

268

5.1.2 Top-coding

To protect the privacy of high-income individuals, income data are often sub- ject to top-coding, that is, the actual income amount is recorded if it falls below a certain threshold, otherwise the survey only contains a flag indicat- ing that the income is top-coded at the threshold value. A leading example are the public-use releases of the US Current Population Survey (see, e.g., http://cps.ipums.org/cps/topcodes tables.shtml).

Sampling is censored if the population proportion of individuals with income below the top-coding threshold is known or may be accurately estimated, otherwise it is truncated.

To analyze the problems caused by top-coding, let the variability of income in the population be represented by a nonnegative continuous random variable Z

⇤, let the income threshold be equa to some constant c > 0, and let D be a binary indicator equal to one if Z⇤ 2 B = [0, c) and equal to zero otherwise, so D = 1[Z⇤ < c]. The fraction of the population for which actual income may be observed is equal to

P[D = 1] = P[Z⇤ < c] = F(c),

where F denotes the DF of Z⇤. Although actual income is unobservable when it exceeds c, measured income is often conventionally set equal to c in this case. With this convention, measured income may be represented by the random variable

Z = DZ⇤ + (1 � D)c = min{Z⇤, c},

corresponding to a right-censored version of Z⇤ with fixed censoring at the point c.

269

Truncated samples

Let the data {(Di, Zi)} n i=1 consist of n independent observations on (D, Z).

A truncated sample only contains individuals with income below the thresh- old c, so Di = 1 for all i.

Conditional on Di = 1, the DF of an observation Zi from a truncated sample is

G(z | Di = 1) = P[Z⇤  z | Z⇤ < c] =

8 <

:

F(z)

F(c) , if z < c,

1, otherwise (5.1)

(Figure 13). Since F(c) is unknown, it is generally impossible to identify F on the basis of a truncated sample. Thus, the EDF of the observed data provides a good estimate of the ratio F(z)/F(c) for z < c, but not of F(z).

270

Figure 13: Population DF F(z) and conditional DF G(z | Di = 1) for a trun- cated sample.

0 .2

.4 .6

.8 1

-2 0 2 z

F(z) G(z|D=1)

271

Censored samples

A censored sample contains individuals with income both below c and above c, so both Di = 0 and Di = 1 are observed. Although the actual income of those with Di = 0 is unknown, you have su�cient information for estimating P[D = 1], for example by D̄ = n�1

Pn i=1 Di.

The conditional DF of Zi given Di = 1 is equal to (5.1), while the DF of a top-coded observation is degenerate with all its mass concentrated at the point c, so G(z | Di = 0) = 1[z � c]. Because the marginal distribution of Di is

P[Di = d] = ( 1 � F(c), if d = 0,

F(c), if d = 1,

the joint DF of (Di, Zi) is

G(D, Z) = G(z | Di = d) P[Di = d]

=

( 1[z � c] (1 � F(c)), if d = 0,

1[z < c] F(z) + 1[z � c] F(c), if d = 1.

The marginal DF of a sample observation Zi is therefore

G(z) = G(z, 0) + G(z, 1) =

( F(z), if z < c,

1, if z � c

(Figure 14), which is the DF of a mixed (continuous-discrete) distribution assigning probability mass 1 � F(c) to the single point z = c and spreading the remaining mass F(c) over the interval (�1, c) according to the DF F(z). Thus, a censored sample contains enough information to identify F(z) for z < c, but no information to identify F(z) for z � c.

It is easily seen that, if µ = E[Z⇤] exists, then the mean of Zi is

E[Zi] = µ � Z 1

c

[1 � F(z)] dz < µ.

Although knowledge of F(z) for z < c is generally insu�cient to identify the mean of Z⇤, it is nevertheless su�cient to recover all the quantiles ⇣p of Z

⇤

such that p  F(c). In particular, it is enough to identify the median of Z⇤

whenever F(c) > 1/2, that is, c is above the median.

272

Figure 14: Population DF F(z) and DF G(z) for a censored sample.

0 .2

.4 .6

.8 1

-2 -1 0 1 2 z

F(z) G(z)

273

5.1.3 Random censoring

In prototypical model of self-selection into the labor force (see e.g. Heckman 1978), people work (D = 1) if their potential market wage Z⇤ exceeds their reservation wage W ⇤, and do not work (D = 0) otherwise. The probability of a person working is therefore

P[D = 1] = P[W ⇤ < Z⇤] = P[C < 0],

where C = W ⇤�Z⇤. The model also assumes that the observed wage coincides with the potential wage for those who work and is conventionally set to zero for those who do not work. The observed wage may then be represented by the nonnegative random variable

Z = D Z⇤ = 1[W ⇤ < Z⇤] Z⇤,

which corresponds to a left-censored version of Z⇤. Here censoring is not fixed but depends on W ⇤, as Z⇤ is only observed when it exceeds W ⇤.

Let the data {(Di, Zi)} n i=1 consist of n independent observations on (D, Z). A

truncated sample only contains people who work, so Di = 1 for all i. If F(z, c) denotes the joint DF of (Z⇤, C), the DF of Zi for those who work is

G(z | D = 1) = P[Z⇤  z | D = 1] = P[Z⇤  z, C < 0]

P[C < 0] =

F(z, 0)

P[C < 0] .

A censored sample contains instead both people who work and people who do not work, so it contains enough information to estimate P[D = 1]. The DF of Zi for those who work is G(z | D = 1), whereas for those who do not work it is degenerate with all its mass concentrated at zero, that is,

G(z | Di = 0) = P[Zi  z | Di = 0] = 1[z > 0].

The joint DF of (Di, Zi) in a censored sample is therefore

G(d, z) = G(z | Di = d) P[Di = d} = ( F(z, 0), if d = 1,

1[z > 0] P[C � 0}, if d = 0.

Given knowledge of G(d, z), you can then compute the marginal DF of observed wages, G(z) = G(0, z)+G(1, z), and compare it to the marginal DF of potential wages, F(z) = limc!1 F(z, c).

274

5.2 Sample selection

Simple random sampling (Section 4.3.1) is of fundamental importance in theo- retical statistics, because it guarantees that the model representing variability at the population level coincides with the model representing variability at the sample level. However, many sample designs of practical importance are far from this ideal.

To formalize the distinction between the models of variability at the population and at the sample level, I represent the population by a random variable Z with density function f(z) and the sample by a collection Z1, . . . , Zn of IID random variables with common density function g(z) = w(z) f(z), where w(z) is a nonnegative function that represents the sampling design.

Because the density function g must integrate to one, the weight function w must satisfy

1 =

Z g(z) dz =

Z w(z) f(z) dz = E[w(Z)].

This formulation assumes that the sample design consists of n independent replications of the same chance experiment, but allows the model of variability to di↵er at the sample and at the population level.

Simple random sampling corresponds to the case when w(z) = 1. A more gen- eral sample design introduces a systematic di↵erence between the probability distribution of a sample value and that of the population. When this occurs, I say that sampling is selective. In particular:

• values of Z such that w(z) > 1 are oversampled;

• values of Z such that 0 < w(z) < 1 are undersampled;

• values of Z such that w(z) = 0 are systematically missing in the data, as with noncoverage.

Section 5.2.1 and 5.2.2 illustrate with two examples.

275

Figure 15: Population density f, density g of a sample observation, and weight function w = g/f representing the sampling design.

0 .5

1 1. 5

2

0 .2 .4 .6 .8 1 z

f(z) g(z) w(z)

276

5.2.1 Response-based sampling

Let Z = (X, Y ) be a random vector, and suppose that the outcome of interest Y is discrete and takes values 1, . . . , S. Also let As = {(x, y): y = s} be the population stratum for which Y = s. For concreteness, X may be a vector of observable individual characteristics and Y may represent the mode of transportation (automobile, bus, train, etc.).

Response-based sampling consists in drawing at random ns units from each stratum As and recording the associated value of X. This sampling scheme is motivated by the fact that sampling bus riders at a bus stop, train riders at a train station or car drivers at a parking lot, and recording their characteristics X, is often simpler and less expensive than interviewing people at their homes.

More generally, whenever the population units are physically clustered on the basis of the alternative they choose, response-based sampling o↵ers economies of scale that may not be possible under other sampling processes.

If !(s) denotes the sampling fraction from the sth population stratum and f(y) denotes the population probability that Y = y, then the probability of observing Y = y in the sample is

g(y) = !(y)f(y)

PS s=1 !(s)f(s)

= w(y) f(y),

where the weight function

w(y) = !(y)

PS s=1 !(s)f(s)

is generally di↵erent from one. Thus, g(y) 6= f(s).

A notable exception is when the sampling fraction !(s) is the same for all strata, which corresponds to self-weighting or proportional allocation.

277

5.2.2 Flow and stock sampling

Let Z be a nonnegative continuous random variable representing the duration of an unemployment spell, and let f, µ and �2 respectively denote the density function, the mean and the variance of Z.

There are two ways of gathering information about the distribution of Z:

• Flow sampling: consists of drawing a random sample from the popu- lation of those who become unemployed during a specified time period.

• Stock sampling: consists of drawing a random sample from the popu- lation of those who are unemployed at a given point in time.

Suppose you observe unemployment durations Z1, . . . , Zn for a random sam- ple of individuals. For simplicity, assume that these data correspond to com- pleted unemployment spells.

Under flow sampling, the density of a completed unemployment spell Zi for a new entrant is the same as the density f(z) of Z.

Next consider stock sampling. Write the completed duration of an unem- ployment spell for the ith person as Zi = Ui + Vi, where Ui is the elapsed duration (the duration up to the time of the survey) and Vi is the successive duration. For the ith person to be registered as unemployed it is necessary that Ui > 0, that is, elapsed duration must be left-censored. It can be shown that, under stock sampling, the density of Zi is

g(z) = z

µ f(z).

In this case w(z) = z/µ, so stock sampling oversamples spells longer than the mean µ and undersamples spells shorter than µ. In turn, this leads to incorrect inferences about the distribution of Z at the population level. For this reason, stock sampling is said to be length-biased.

For example, under stock sampling, the mean of the observed durations is

E[Zi] = Z 1

0

z g(z) dz = µ

✓ 1 +

� 2

µ2

◆ > µ.

Thus, stock sampling leads to an upward biased measure of mean unem- ployment duration and the relative bias (E[Zi] � µ)/µ is proportional to the squared coe�cient of variation �/µ of Z.

278

5.3 Nonresponse

Nonresponse is a typical problem with microeconomic surveys. Because of nonresponse, the available data do not conform to the ideal case of com- plete data for which standard statistical methods and software packages are typically designed.

Nonresponse not only implies an e�ciency loss relative to the complete- data case, due to the smaller e↵ective sample size, but also makes it harder to identify population parameters. As shown by Horowitz and Manski (1998), the seriousness of the problem is directly proportional to the amount of nonresponse.

It is useful to distinguish between three types of nonresponse:

• unit nonresponse (Section 5.3.1), which occurs when no information is available from a sample unit;

• attrition and new entry (Section 5.3.2), which is a special case of unit nonresponse arising in panel data;

• item nonresponse (Section 5.3.3), which occurs when no information is available from a sample unit on a particular item included in the survey.

279

5.3.1 Unit nonresponse

Unit (or complete) nonresponse occurs when no information is available from a sample unit. It may be caused by refusal or inability of the unit to participate, or by a missing survey instrument.

There is a trend towards increasing unit nonresponse rates in micro-level sur- veys (see, e.g., Meyer, Mok and Sullivan 2015).

I will now consider separately three types of unit nonresponse with somewhat di↵erent characteristics:

• noncontact, i.e. failure to deliver the survey request because of failure to locate the sample unit, noncontact, etc.;

• inability to participate, e.g. a contacted person cannot understand the language of the questionnaire;

• refusal to participate, i.e. the survey request is delivered but is re- jected.

280

Noncontact

Many factors influence whether sample units can be contacted. The situation is di↵erent between face-to-face and telephone surveys on the one hand, and mail, e-mail and web surveys on the other hand.

In face-to-face and telephone surveys, it is not known exactly when people are at home and accessible. So interviewers are generally instructed to make multiple contact attempts on a unit until contact is achieved. Notice that:

• contact attempts in the evening and on weekends are more successful than at other times;

• households where someone is almost always at home are the easiest to contact;

• the percentage of successful attempts tends to decline with each succes- sive attempt.

Mail, e-mail and web surveys make the survey request accessible to the sam- ple unit continuously after it is sent. In these cases, it is more di�cult to di↵erentiate between noncontact (the request has not been seen) and refusal to participate (the request has been seen and rejected).

Inability to participate

Sometimes, sample units are successfully contacted and would be willing to cooperate, but cannot for a number of reasons. For example:

• cannot understand the language(s) of the survey;

• are incapable of understanding the questions or retrieving from memory the information requested;

• have physical health problems;

• do not have the information requested (this sometimes occurs in business surveys).

281

Refusal to participate

Factors influencing refusal to participate include:

• Social environment: in household surveys, large urban areas tend to generate more refusals, while households with more than one member tend to generate less refusals than single person households.

• Personal characteristics: men tend to generate more refusals than women.

• Interviewer characteristics: more experienced interviewers tend to obtain lower refusal rates.

• Survey design: rewards (monetary or in-kind) tend to reduce refusal rates.

While the first two factors are outside the control of a researcher, the last two can be manipulated to increase response rates. See Groves, Cialdini and Couper (1992) and Groves and Couper (1998) for more detail.

Theories of refusal behavior (see Groves et al. 2004, pp. 176–177):

• The social exchange theory, or opportunity cost hypothesis (see also Hill and Willis 2001), posits a “social exchange” between the survey interviewer, who desires information that the respondent has, and the re- spondent, who decides how much information to reveal. People refuse to participate if the act of participation involves costs (actual or perceived) that exceed the expected rewards.

• The leverage-salience theory: di↵erent people attach di↵erent impor- tance to features of the survey, such as topic, burden, incentives, author- ity of sponsor, etc. When a sample person is approached, one or more of these features are made salient in the interaction with the interviewer or with the survey material provided to the person. Depending on what is made salient and how the person values these features, the result could be a refusal or an acceptance.

282

5.3.2 Panel attrition and new entry

A panel is called balanced if all sample units are observed at exactly the same times. Otherwise, it is called unbalanced. Unbalanced panels are the most common case.

Attrition and new entry are typical sources of unbalancedness. Thus, the study of attrition and new entry plays a key role in evaluating what you can learn from a panel survey about population characteristics of interest.

Attrition

Attrition arises when units initially included in a panel survey are lost through time. Attrition a↵ects, more or less seriously, all microeconomic panels.

An important distinction is between exogenous and endogenous attrition:

• attrition is exogenous if it is unrelated to the outcome of interest Y ; exogenous attrition does not bias the information carried by the sample about the population regression function of Y and typically only causes e�ciency problems;

• attrition is endogenous if it is systematically related to Y ; endogenous attrition may cause bias if not properly controlled for.

For example, in panels of individuals, attrition is typically associated with important transitions in a person’s life: going to college, finding a new job, marriage or divorce, retirement, etc. If these events are the main object of study, then attrition is endogenous. This may lead to invalid inference about the population of interest, even when attrition rates are modest.

Attrition depends on both individual behavior and the procedures followed by the agency collecting the data. This distinction is important but often ignored.

It is also useful to distinguish between intermittent and monotone attri- tion, depending on whether or not a panel participant, who at some point leaves, returns to the panel at a later date. In the first case units are lost permanently, in the second they are lost only temporary.

283

New entry

It is another kind of unit nonresponse arising in panel surveys, and is defined as failure to obtain data from a sample unit at any wave before its addition to the initial sample.

Depending on its characteristics, new entry may reduce or exacerbate the e↵ects of attrition on the representativeness of a panel.

284

5.3.3 Item nonresponse

Item (or partial) nonresponse occurs when a unit refuses to answer, or fails to provide a useful response, to a particular question included in the survey.

Qualitatively, the impact of item nonresponse is the same as that of unit nonresponse. There are, however, important di↵erences:

• The damage tends to be limited when item nonresponse is confined to a few variables, but can be substantial when the cumulative e↵ect of item nonresponse to several related questions is considered (e.g., the e↵ect on total income of item nonresponse to questions about specific income components).

• The causes of item and unit nonresponse tend to be di↵erent. Among those that have been studied are:

– inadequate comprehension of the intent of a question;

– failure to retrieve adequate information;

– lack of willingness or motivation to disclose the information requested.

The rate of item nonresponse tends to di↵er across variables, and is typically found to be highest for questions on income and assets.

285

5.4 Missing-data mechanisms

In microeconomic data, nonresponse is one of the main reasons why data may be missing. As Kline and Santos (2013) argue:

“Despite major advances in the design and collection of survey and administrative data, missing and incomplete records remain a pervasive feature of virtually every modern economic data set.”

To formalize the general missing data problem, let M be a missing data indicator, equal to one if the information on some outcome of interest Y is missing and to zero otherwise, and let X be a vector of covariates that may help predict missingness and are always observed. The available data consist of a sample {(Mi, Xi, Yi)}

n i=1 from (M, X, Y ), with Yi missing if Mi = 1.

What variables may be included in X depends on the specific missing data problem. For example, in the case of unit nonresponse the only available information is from the frame and the fieldwork. In the case of item non- response one has the additional information on the nonmissing variables. In the case of attrition one also has the information from previous waves.

Let f(m, z) = f(m, x, y) denote the joint density of (M, Z) = (M, X, Y ) and let

p(m | z) = P[M = m | Z = z], m = 0, 1, denote the conditional probability of M.

Following Rubin (1976), Little and Rubin (2002) and Seaman (2013), I dis- tinguish between three types of missing-data mechanisms depending on the nature of the joint density f(m, z) or, equivalently, on the nature of the con- ditional probability p(m | z). The equivalence between the two formulations follows from the fact that

f(m, y | x) f(x) = f(m, z) = p(m | z) f(z).

286

5.4.1 Missing completely at random

Data are missing completely at random (MCAR) if M does not depend on the variables in Z that is, neither on X nor on Y , also written M??Z or M??(X, Y ). Thus, MCAR is equivalent to the condition that

f(m, z) = f(z) p(m),

or that p(m | z) does not depend on z.

5.4.2 Missing at random

Data are missing at random (MAR) if, after conditioning on the co- variates in X, there is no dependence between M and Y , also written (M??Y ) | X. Thus, MAR is equivalent to the condition that

f(m, y | x) = f(y | x) p(m | x),

Because it is always true that

f(m, y | x) = p(m | x, y) f(y | x) = f(y | m, x) p(m | x),

MAR implies that f(y | m, x) = f(y | x)

and p(m | x, y) = p(m | x).

MAR does not hold if either of these two conditions fails.

5.4.3 Missing not at random

Data are missing not at random (MNAR) if, even after conditioning on the covariates in X, there is dependence between M and Y , that is,

f(m, y | x) 6= f(y | x) p(m | x).

Equivalently, data are MNAR if p(m | z) depends on y and possibly also on x.

287

5.4.4 Ignorable and nonignorabile missingness

A missing-data mechanism is ignorable if data are MCAR or MAR, and is nonignorable if data are MNAR.

The dominant approach to missing data assumes ignorability. However, as argued by Kline and Santos (2013), this is

“. . . an assumption whose popularity owes more to convenience than plausibility. Even in settings where it is reasonable to be- lieve that nonresponse is approximately ignorable, the prevalence of missing values in modern economic data suggests that economists ought to assess the sensitivity of their conclusions to small devia- tions from this assumption.”

In what follows I shall consider three di↵erent approaches to missing data:

• Design the survey to reduce nonresponse (Section 5.4.4). This ex-ante approach is obvious but may be costly and is unfeasible if the data have already been collected.

• If the missing data mechanism is ignorable, ex-post approaches include:

– reweighing the data to compensate for the units lost to nonresponse (Section 5.5.1);

– replacing the missing values with imputations based on a statis- tical model (Section 5.5.2).

• If the missing data mechanism is nonignorable, ex-post approaches include:

– modeling missingness explicitly (Section 5.6.1),

– measuring the uncertainty that missingness creates (Section 5.6.2).

288

Survey design to reduce missingness

A number of aspects of survey design can influence the probability of missing data and may therefore be adjusted to reduce missingness. These include:

• Survey content: a survey on drugs or on financial matters needs careful ordering and wording of the questions, or some randomized response technique.

• Design of the survey instrument: very important. For example, the complexity of a questionnaire is known to negatively a↵ect response rates and data quality. Important dimensions of complexity are: (i) how long it takes to fill the questionnaire, (ii) the wording of the questions, (iii) the reference period for the questions, and (iv) the branching and skip patterns.

• Characteristics and training of the interviewers: they matter a lot.

• Data collection methods: telephone, mail, fax and internet surveys tend to have lower response rates than in-person surveys.

• Time of the survey: vacations, weekends, time of the day may matter.

• Survey introduction: the respondent should be told clearly what the purpose of the survey is and should be assured about anonimity and confidentiality.

• Rewards: monetary and nonmonetary rewards appear to increase co- operation, but their e↵ect on data quality is not clear.

289

5.5 Approaches under ignorability

5.5.1 Weighting

Surveys with complex design usually display unequal selection probabil- ities, di↵erent response rates across subgroups, and departures from known distribution of key variables (e.g. gender) for the target population. To compensate for these features, survey agencies typically provide survey weights.

There are four main types of weights:

• First-stage ratio adjustment weights w (1) i (for stratified multistage

designs): To stabilize estimates over di↵erent selections of PSUs in a stratum. They are usually equal to the ratio between the population size of a stratum (from the sampling frame) and its survey estimate.

• Base (design) weights w (2) i : For di↵erential selection probabilities.

They are usually equal to the inverse of the selection probabilities.

• Nonresponse weights w (3) i : To adjust for unit nonresponse. They

are usually equal to the inverse of the (estimated) probabilities of unit nonresponse.

• Post-stratification weights w (4) i : To ensure that survey estimates of

sub-totals (e.g. by gender or age group) agree with external sources.

Both the first-stage ratio adjustment weights and the post-stratification weights are typically based on information from the target population, not on sample information.

Once the various weights have been computed, they are combined into a final weight

wi = w (1) i w

(2) i w

(3) i w

(4) i , i = 1, . . . , n.

Often, surveys only provide the final weights, not their individual components.

290

Estimates of population size, population total and population mean

Using the final weights, an estimate of the population size is

N̂w = nX

i=1

wi,

an estimate of the population total of Z is

⌧̂w = nX

i=1

wiZi,

and the resulting estimate of the population mean of Z is

µ̂w = ⌧̂w

N̂w

=

Pn i=1 wiZiPn i=1 wi

.

Notice that µ̂w does not change if the weights are rescaled up or down. Thus, for estimation of the population mean, it does not matter if the weights add up to the total size of the population, or they add up to one.

All three estimators are constructed by analogy with the estimators presented in Section 4.3.4. Because of the complex nature of the final weights, how- ever, their sampling properties tend to be quite complicated, especially if the missing-data mechanism is nonignorable.

In general, weights based on sample information tend to increase the sampling variance of survey estimators, but are used in order to reduce biases resulting from noncoverage and nonresponse.

On the other hand, weights using frame population information (such as the first-stage adjustment weights and the post-stratification weights) tend to reduce the sampling variance of survey estimators.

291

5.5.2 Imputations

Survey weights only adjust for unit nonresponse. Adjusting for item nonre- sponse is typically done through imputation. This procedure consists of filling- in values that are missing, either because of item nonresponse or because the response is considered incorrect or implausible.

When a missing value is imputed, the dataset should in principle be supple- mented by an indicator (or “flag”) showing whether the response was measured or imputed. Sometimes, this is not done.

Imputing missing values is often advocated on the ground that:

• it makes the data easier to analyze, for standard methods (e.g. OLS) may then be applied to the completed data;

• it reduces nonresponse bias relative to the alternative of excluding all observations with missing values.

In some cases, imputation may indeed reduce the bias relative to the case where missing values are simply ignored. However, as pointed out by Dempster and Rubin (1983):

“the idea of imputation is both seductive and dangerous. It is se- ductive because it can lull the user into the pleasurable state of believing that the data are complete after all, and is dangerous be- cause it lumps together situations where the problem is su�ciently minor that it can legitimately be handled in this way and situations where standard estimators applied to real and imputed data have substantial bias”.

A key problem with the imputation methods discussed in the next two sections is that they all rely on the assumption that the data are MAR, conditional on the auxiliary variables employed in the imputation procedure. This assumption is untestable unless external information is available, such as administrative records or a follow-up of nonrespondents.

If the MAR assumption does not hold, then other imputation methods should be considered that explicitly model the nonresponse mechanism (Greenlees, Reece and Zieschang 1982).

292

Deterministic imputations

I distinguish between deterministic and random imputation methods.

Deterministic imputation methods replace missing values with means or other non-random values. Examples include:

• Mean imputation: Replaces the missing value of a variable Y for a nonrespondent with the mean value of Y for the respondents.

• Cell mean imputation: The sample is first partitioned into cells, based on the values of a set of auxiliary variables Z. Then the missing values of Y in a cell are replaced with the mean value of Y for the respondents in that cell.

• Hot-deck imputation: Replaces the missing value of Y for a nonre- spondent (or recipient) with the value of Y for a “similar” respondent (or donor).

• Regression imputation: Replaces the missing values of Y with the predicted values from a regression of Y on a set of auxiliary variables Z using the nonmissing data. Mean imputation is the special case when Z only contains the constant term. Regression imputation requires that all auxiliary variables in Z have nonmissing values. When this is not the case, then the missing auxiliary variables are imputed through some preliminary imputation procedure.

• Cold-deck imputation: Replaces the missing value of Y with values from a previous survey or a di↵erent data set, e.g. historical data.

Hot deck methods are popular (for example, they are used by the Bureau of Census for imputing missing values in the US Current Population Survey) because they do not rely on model fitting and replace missing values with values that are “plausible” become they come from respondents. See Lillard, Smith and Welch (1986), Andridge and Little (2010), and Bollinger et al. (2015) for more details.

The main problem with deterministic imputation is underestimation of the variability of the variable of interest.

293

Random imputations

These methods replace missing values with randomly chosen values and are designed to avoid underestimation of the variability of the variable of interest.

Examples include:

• Random mean imputation: Replaces the missing values of Y with random draws from a N(Ȳ , s2) distribution, where Ȳ and s2 are the sample mean and variance of Y for the respondents.

• Random cell mean imputation: Replaces the missing values of Y with random draws from a N(Z̄cR, s

2 cR) distribution, where Ȳc and s

2 c are

the sample mean and variance of Y for the responding units in cell c.

• Random regression imputation: Replaces the missing values of Y with the predicted value from an auxiliary regression of Y on a set of auxiliary variables Z using the nonmissing data, plus a randomly gen- erated error term (typically, a random draw from the set of residuals of the auxiliary regression).

• Random hot-deck imputation: A donor i is selected at random from the set of respondents (or donor pool) and the imputed value of Y for the jth recipient is computed as Y ⇤j = Yi. The imputed value Y

⇤ j is

therefore a random draw from the EDF of Y for the respondents. To preserve key covariances, the same donor may be used for imputation of related missing variables.

• Random hot-deck imputation within classes: The sample is first partitioned into a number of “imputation classes” defined on the basis of auxiliary variables Z. A donor i is then selected at random from the set of respondents within the class containing the jth case, and the imputed value of Y for the jth recipient is computed as Y ⇤j = Yi. The imputed value Y ⇤j is therefore a random draw from the edf of Y within the class. To preserve key covariances, the same donor may be used for imputation of related missing variables within the class.

294

When are imputations adequate?

In general, survey statisticians try to include in the imputation all the relevant explanatory variables (see Schafer 1997). Unfortunately, considering all the variables that are relevant may impractical because of multicollinearity or degrees of freedom problems, or just impossible.

This means that imputation procedures necessarily impose exclusion restric- tions that may be at odds with the restrictions imposed by applied researchers in their model of interest.

I suggest checking whether your model of interest includes any relevant re- gressor which has been omitted from the imputation procedure. If all your regressors are used as auxiliary variables in the imputation, then I suggest using the imputed data, possibly correcting the estimator’s variance using multiple imputation methods.

A second problem arises when the model of interest involves a nonlinear transformation of the outcome variable (e.g., the log transformation or a cat- egorization). Thus, let D be a binary indicator equal to 0 if Y is missing, let Y

⇤ denote the imputed value of Y , and let g(·) be a nonlinear transformation. An imputation such that

E[Y ⇤ | D, Z = 0] = E[Y | Z]

does not imply that

E[g(Y ) | Z] = E[g(Y ⇤) | D, Z = 0].

So, for example, if the imputation model for Y is a linear regression, a model involving a monotone transformation of Y may lead to inconsistent estimates. More generally, if the imputation model is linear in Y , then an imputed data estimator based on a moment function (.) that is nonlinear in Y may be inconsistent.

Following Meng (1994), I say that the model of interest and the imputation models are uncongenial if they are based either on di↵erent sets of ex- planatory variables or on di↵erent parametric assumptions. If the two models are uncongenial, biases are likely to arise.

295

Multiple imputations

When imputed values are treated as “real values”, standard errors of estimates are underestimated, leading to confidence intervals that are too narrow and test statistics that are too large.

Multiple imputation (Rubin 1987) works as follows:

• Define an imputation model for the relationship between the poten- tially missing Y variables and a vector of regressors X.

• Estimate the imputation model using the nonmissing data.

• Replace the missing values with random draws from the estimated im- putation model, thus obtaining a completed data set. Repeat m � 2 times.

• Analyze each of the m completed datasets by standard complete-data methods and obtain m estimates of the statistics of interest.

• Take the sample variance of these m estimates as a measure of the un- certainty due to the presence of missing values.

296

5.6 Approaches under nonignorability

I first show what bias may arise assuming ignorability (i.e., MCAR or MAR) when the missing data mechanism is in fact nonignorable. Let Zi = (Xi, Yi) and suppose that the complete data obey the classical linear model

Yi = ↵ + �Xi + Ui, i = 1, . . . , n, (5.2)

where Ui has zero mean, finite variance and is mean independent of Xi. Thus, if there were no missing data, an OLS regression of Yi on a constant and Xi would give unbiased and consistent estimates of ↵ and �.

Now consider the OLS estimator �̂ in a regression of Yi on a constant and Xi using the available nonmissing data, namely those with Mi = 0. There are two problems with this estimator of �:

• The e↵ective sample size is n(1 � M̄) < n, where M̄ = n�1 Pn

i=1 Mi

is the fraction of missing data, so �̂ is less e�cient than the unfeasible estimator based on the complete data.

• More importantly, the model for the conditional mean of the available nonmissing data is not (5.2) but

E[Yi | Xi, Mi = 0] = ↵ + �Xi + hi, i = 1, . . . , n,

where hi = E[Ui | Xi, Mi = 0]. Thus, �̂ may su↵er of omitted variable bias due to omission of the term hi. Specifically, �̂ is biased and incon- sistent for � unless missingness is ignorable, that is, one of the following two conditions holds:

– hi = 0; this is the case when E[Ui | Xi, Mi = 0] = E[Ui | Xi], that is, Ui is mean independent of Mi conditionally on Xi;

– hi 6= 0, but Xi and hi are uncorrelated.

297