Week 12: Experimental Design
Sample Study
- Estimate the parameters of the population
- Inferential statistics to determine
- Make sure the sample is representative of the population before
analyzing it
Observational Study
- How different parameters in the population behave together
- Draw conclusions on correlations
- No outside intervention during the study (use data available)
- EX: effect of drinking tea before bed, take random sample and ask
questions
- Benefits: examine long-term effects, avoid unethical experiments,
some experiments not possible, new methods and tech for causal
estimations
Experimental Study
- Establish causality from observational study in a controlled
environment
- Design an experiment to study a certain effect by intervention
- Plan for data before you collect it
- EX: create controlled environment, give one group tea before bed and
other group does not, compare results
- Benefits: direct comparison between treatments of interest, minimize
bias, error in comparison is small, in control of experiments (make
stronger inferences about nature of differences and make inferences
about causation)
Confounder
- Extraneous variable in observational study that correlates with
dependent and independent variables
- EX: breastfeeding linked to higher IQ but association could be due to
socioeconomic status (women who breastfeed tend to be better
educated)
Spurious correlation
- Relationship between two variables appear to have interdependence
or association with each other but actually don’t
Controlled experiment
- Control = only reliable way to measure response to changing
variables
- Estimation = guarantees learn something about what you want to
know
- Efficiency = learn the most from the experiment
Experimental design
- Process of planning a study to meet specified objectives
- Planning an experiment properly is very important - ensure right type
of data and a sufficient sample size and power are available to answer
research questions of interest as clearly and efficiently as possible
- Planning = What measurement to make (response)? What condition to
study (treatment)? What experimental materials to use (units)?
Step 1 to experiment
- Make control and treatment groups
- Blind experiment = sample don’t know difference between
groups
- Double blinded = researches don’t know difference
Placebo effect
- Sample knows about experiment and think about results
- To avoid -control group given the same looking test
Step 2 to experiment
- Statistical analysis to see if difference is significant
Step 3 to experiment
- Test replicability of experiment
- Might have done experiment under a certain weather condition that
may influence the result
- Sample may be from a certain race and culture
- Let others be skeptical about results and repeat experiment to see if
having same outcome as yours
Terms of experimental design
- Treatments = different procedures we want to compare
- Experimental units = apply the treatment to
- Responses = outcomes we observe after applying treatment to
experimental unit
- Randomization = known, understood probabilistic mechanism for
assignment of treatments to units
- Experimental Error = random variation present in all experimental
results
- Measurement units = actual objects the response is measured
- Blinding = evaluator doesn’t know which treatment was given to
which unit
- Control = standard treatment used as a baseline for the other
treatment
- Placebo = null treatment used when act of applying a treatment, any
treatment, has an effect
- Factors = combine to form treatment
- Levels of the factor = individual setting for each factor
- Block = arranging of experimental units in groups (blocks) similar to
one another
- Confounding = effect of one factor or treatment can’t be distinguished
from another factor or treatment
A/B testing (split or bucket testing)
- Comparing two versions of a webpage or app against each other to
determine which one performs better
- Two or more variants of a page shown to users at random
- Statistical analysis used to determine which performs better for
conversion goal
Process of A/B testing
- Take webpage/app screen and modify it to create a second version of
same page
- Change you want to see should be controlled to a single change
- Use a script to randomly show half of your visitors original version
(control) and other half are exposed to modified version (variation)
A/B testing tools
- Optimizely, Visual web optimizer, Adobe target, Google content
experiments
-
Powered by TCPDF (www.tcpdf.org)
