Reflection paper Applied Business Analytics

Predictive Analytics for Business Strategy: Reasoning from Data to

Actionable Knowledge By Jeffrey T. Prince

Chapter 4: The Scientific Method: The Gold Standard for Establishing Causality

The Scientific Method The scientific is ap process designed to generate knowledge through the collection and analysis of

experimental data.

The full process for the scientific method consisting of the following six parts:

1. Ask a question – e.g., “What can reduce the likelihood of cancer?”

2. Do background research. Review the literature that addresses the nature of your question…

This will help you determine novel research opportunities that address your question of interest.

3. Formulate a hypothesis, or a possible answer, to your research question. A hypothesis is a

proposed idea based on limited evidence that leads to further investigation.

4. Run an experiment! An experiment is a test within a controlled environment designed to

examine the validity of a hypothesis. A treatment w/n an experiment is something that is

administered to members of at least one participating group (usually, an “experimental” group

will get the treatment and a “control” group will not get the treatment). Finally, the treatment

effect is the effect of the main independent variable under analysis – i.e., the change in the

outcome (the dependent variable) resulting from a treatment, or the change in a treatment

variable measured as 1 if you get the treatment and 0 otherwise.

5. Next, we must analyze the data from the experiment and draw conclusions. In other words, we

simply compare the measured outcomes between the group receiving the treatment and the

group that did not.

6. The final step is to communicate the findings – i.e., dissemination, policy implications, and areas

for future research.

Here is a channel where you can learn all about FUN science experiments:

https://www.youtube.com/watch?time_continue=161&v=ugRc5jx80yg

The Scientific Method and Causal Inference The basic goal when running an experiment is to measure a treatment effect – i.e., the change in the

outcome resulting from variation in the treatment (e.g., whether you get it).

The treatment effect must be able to obtain the counterfactual – i.e., answer the question “What is the

outcome for an individual who does not receive the treatment?”

There are two ways we can measure the “counterfactual” – first, compare measurements between

groups, and, second, compare measurements over time.

Using the potential outcomes framework, consider a group of subjects who will participate in an

experiment, indexed w/ the letter 𝑖 – e.g., 𝑖 = 1 is the first subject, 𝑖 = 2 is the second, and so on.

For any given subject 𝑖, consider the outcome realized by that subject if it receives the treatment (𝑇).

We denote this as 𝑶𝒖𝒕𝒄𝒐𝒎𝒆𝒊 𝑻. Similarly, consider the outcome realized by that subject if it does not

receive the treatment (𝑁𝑇). We denote this as 𝑶𝒖𝒕𝒄𝒐𝒎𝒆𝒊 𝑵𝑻.

The treatment effect for subject 𝑖 is simply the difference:

𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝐸𝑓𝑓𝑒𝑐𝑡𝑖 = 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑇 −𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖

𝑁𝑇

Since we are unable to measure treatment effects for individual subjects, we attempt to measure the

mean, or average, treatment effect.

The average treatment effect (ATE) is the average difference in the treated and untreated outcome

across all subjects in a population.

In measuring the ATE, we consider each subject’s treatment effect as a draw from the population of

treatment effects across all possible subjects.

Consequently, 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝐸𝑓𝑓𝑒𝑐𝑡𝑖 is a random variable whose distribution mirrors the distribution of

treatment effects across the entire population of possible subjects.

The average treatment effect is simply the expected value of the treatment effect for a randomly drawn

subject from the population, written as

𝐸[𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝐸𝑓𝑓𝑒𝑐𝑡𝑖].

Expanding on this, we have:

𝐴𝑇𝐸 = 𝐸[𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝐸𝑓𝑓𝑒𝑐𝑡𝑖] = 𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑇 −𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖

𝑁𝑇]

How can experiments help us estimate the ATE?

The answer to this question is best understood by considering an experiment w/ a dichotomous

treatment – that is, a treatment in which participants are split into two groups where one receives the

treatment (takes a drug) and the other does not (takes a placebo).

To begin, we define two more variables, whose values are determined by the experiment.

The first is a dichotomous variable, 𝑻𝒓𝒆𝒂𝒕𝒆𝒅𝒊. This variable equals 1 if subject 𝑖 actually received the

treatment during the experiment, and 0 if the subject did not receive the treatment.

The second is 𝑶𝒖𝒕𝒄𝒐𝒎𝒆𝒊. This variable equals the outcome actually experienced by subject 𝑖 after the

experiment.

With these definitions, note that:

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 = 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑇 𝑖𝑓 𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 1

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 = 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑁𝑇 𝑖𝑓 𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 0

It may seem unnecessary to create the variable 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖. However, the values for 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑇 and

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑁𝑇 do not depend on which group the subject was assigned to.

Rather, they are the outcomes that the subject would realize when treated or untreated, regardless of

the group to which the subject was actually assigned (like the population parameter for the effect).

In contrast, 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 does depend on the group assignment; it is the outcome actually realized, which

may depend on whether the treatment was actually received or not.

For example, suppose my cholesterol level falls 21 points if I take an experimental drug but remains

unchanged if I do not take the drug.

Further, suppose I was given the drug in the experiment – i.e., I was assigned to the treatment group.

Then, 𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 1, 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑇 = −21, 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖

𝑁𝑇 = 0, and 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 = −21.

If instead I was not given the drug, we would then have 𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 0 and 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 = 0; the other

two variables would not change – i.e., 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑇 = −21 still and 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖

𝑁𝑇 = 0 still.1

When we run an experiment, it will generate sample data consisting of realizations for 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 and

𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 for all subjects.

Using these data, we can calculate the mean outcome for those in the treated group

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅ and the mean outcome for those in the untreated group

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅.

For our drug example, we can calculate the average change in cholesterol level for those receiving the

drug, and then the average change in cholesterol level for those who did not receive the drug.

Question: When does the difference in the mean outcomes across the treated and untreated groups

yield an unbiased estimate of the ATE, that is:

𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅−𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅] = 𝐴𝑇𝐸

Answer: If we take the difference in these two measures, we can get an estimate of the ATE if the

following two conditions are satisfied:

1. Participants are a random sample of the population.

2. Assignment into the treated group is random.

If participants are a random sample of the population, then means for the data sample are unbiased

estimators for their population counterparts. This relationship is true for conditional means as well.

Thus, in our case, we have:

𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅] = 𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖| 𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 1] and

𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅] = 𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖| 𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 0]

In words, the mean outcomes for those assigned the treatment and those that weren’t in the

experiment are unbiased estimates of the mean outcomes in the population for those that would have

received the treatment and those that wouldn’t.

It follows that:

𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅−𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅] =

1 Or, the effect of the treatment for the ith person who gets it remains the same – i.e., the effect of the treatment overall does not depend on whether a single person gets it (the TE is what the TE is).

𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖| 𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 1]−𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖| 𝑇𝑟𝑒𝑎𝑡𝑒𝑑𝑖 = 0]

That is, the expected difference in the mean outcome for the treated group and untreated groups

equals the difference in the mean outcome in the population b/w those that would have received the

treatment and those that wouldn’t.

Finally, it can be shown that w/ random assignment, the ATE does not suffer from selection bias, which

occurs when the treated and untreated groups start from different places.

Hypothesis Testing for the Treatment Effect Suppose a firm is interested in determining whether a higher clickthrough rate on one of their products

justifies the expense associated w/ purchasing a top-four search result from a search engine.

The process of conducting a hypothesis test requires us to state a null hypothesis to be tested. We

would like to know whether an effect exists, so we might state the null as:

A change in ad position from fourth to first affects click-through rates.

When we use data to reason whether the null is true, we can only make strong statements when the

data cause us to reject the null – e.g., we reject the null w/ 95% confidence.

In contrast, if we instead fail to reject the null, we merely fail to find evidence against it, which is only

weak support for it.

We can make a much stronger argument that there is an effect of ad positioning by rejecting the claim

that there is not an effect, as opposed to failing to reject that there is one.

To make a strong statement about there being an impact of ad placement, our null should be:

H0: Changing an ad’s placement from 4th to 1st has (on average) no impact on click-through rates.

If we define the incidence of a click as the outcome and the movement of an ad from 4th to 1st as the

treatment, the null can be rewritten as follows:

𝐻0:𝐴𝑇𝐸 = 𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑇 −𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖

𝑁𝑇] = 0

The data from our experiment will provide us w/ the average incidence of a click for ads in top position

and the click through rate for ads in 4th position.

Hence, the data will provide us w/ 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅ and 𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅.

We know that, for a large, random sample, these sample means have the following distributions:

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅~𝑁(𝜇1, 𝜎1

√𝑁1 )

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅~𝑁(𝜇0, 𝜎0

√𝑁0 )

Here we define: 𝜇1 = 𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1], 𝜇0 = 𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0], 𝜎1 =

√𝑉𝑎𝑟[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1], 𝜎0 = √𝑉𝑎𝑟[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0], 𝑁1= number of treated

observations, and 𝑁0= number of untreated observations.

A well-known property in stats is that the sum, or difference, of normal random variables is also a

normal random variable.

As a result, we know that the difference in the click-through rates b/w the treated and untreated

observations is normally distributed, since each click-through rate is normally distributed.

Using this fact along w/ basic formulas for expected value and variance, we now have:

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅−𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅~𝑁(𝜇1 −𝜇0,√ 𝜎1 2

𝑁1 + 𝜎0 2

𝑁0 )

Lastly, we know that random assignment implies 𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅]−

𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅] = 𝐴𝑇𝐸 and

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅−𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅~𝑁(𝐴𝑇𝐸,√ 𝜎1 2

𝑁1 + 𝜎0 2

𝑁0 )

If we add our null that the treatment effect equals 0, we have:

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅ −𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅~𝑁(0,√ 𝜎1 2

𝑁1 + 𝜎0 2

𝑁0 )

We can now calculate our test statistic as follows (recall that our test statistic measures, for a single

draw of a random variable, the number of standard deviations that draw is from the mean (or, in the

case of hypothesis testing, from the hypothesized value of zero effect)):

𝑡 =

(

𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 1̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅−𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖|𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 = 0̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅̅ ̅−0

√ 𝑆1 2

𝑁1 + 𝑆0 2

𝑁0 )

We have replaced the population standard deviations 𝜎 with sample standard deviations (S).

Once we have our calculated t-state, we can compare it to the crucial t-stat to determine whether we

should reject or fail to reject the null.

Example 1:

Given the following data, test the following hypothesis:

𝐻0:𝐴𝑇𝐸 = 𝐸[𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖 𝑇 −𝑂𝑢𝑡𝑐𝑜𝑚𝑒𝑖

𝑁𝑇] = 0

Year Profits (Millions)

Number of searches w/ ad in top position 49872

Number of searches w/ ad in 4th position 50128

Click-through rate for top position ad 0.0782

Click-through rate for 4th position ad 0.072415

Standard deviation of clicks for ad in top position 0.268486

Standard deviation of clicks for ad in 4th position 0.259173

You can enter the following data in Stata by first clicking on the “statistics” tab, then select “summaries,

tables, and tests,” then select “Classical tests of hypotheses,” and then select the “t-test calculator.”

Then, simply check the appropriate options and input data as seen here:

Then, select “OK” to obtain the following results:

Pr(T < t) = 0.9997 Pr(|T| > |t|) = 0.0005 Pr(T > t) = 0.0003 Ha: diff < 0 Ha: diff != 0 Ha: diff > 0

Ho: diff = 0 degrees of freedom = 99998

diff = mean(x) - mean(y) t = 3.4666

diff .005785 .0016688 .0025142 .0090558

combined 100,000 .0753001 .0008344 .2638732 .0736646 .0769356

y 50,128 .072415 .0011576 .259173 .0701461 .0746839

x 49,872 .0782 .0012022 .268486 .0758436 .0805564

Obs Mean Std. Err. Std. Dev. [95% Conf. Interval]

Two-sample t test with equal variances

You could have also entered the following line of code into Stata’s command prompt:

. ttesti 49872 0.0782 0.268486 50128 0.072415 0.259173

As you can see, the calculated t-value is equal to 3.4666, which is greater than the critical t-value of

1.96.

Hence, we reject the null hypothesis of “no difference,” or “no average treatment effect,” that is we

have statistical evidence that ad position matters!

The 95% confidence interval for the “difference” is also provide by Stata. It is equal to 0.0025142 to

0.0090558.

Since the value for the hypothesized difference equal to 0 is not contained w/n the confidence interval,

we can again reject the null hypothesis of a 0 treatment effect, or 0 ATE.