Jupyter Notebook
David.sumra
Slides.pdf
Moderation in Marketing Mix Models
Chapter 15
● Understand segments in marketing ● Create dichotomous variables ● Include categorical variables in marketing mix models ● Appreciate a marketing mix model with moderation ● Design a marketing mix model with moderation ● Analyze interaction results of a marketing mix model with moderation ● Analyze follow-up results of a significant interaction with a dichotomous moderator
variable ● Advanced (optional): Analyze follow-up results of a significant interaction with a
continuous moderator variable using spotlight analyses
Learning Objectives
Marketing Analytics Brennan Davis
Dichotomous Variables
● An dichotomous variable is 1 if the case falls into the category represented by the dichotomous variable, and 0 otherwise
● Take one categorical variable, and replace it with several dichotomous variables
● If a variable is categorical (e.g., word categories like region of the U.S.), then create one dichotomous variable for each category of the variable
5
State Region South West Northeast Midwest
Alabama South 1 0 0 0
Alaska West 0 1 0 0
Arkansas South 1 0 0 0
California West 0 1 0 0
Colorado West 0 1 0 0
Connecticut Northeast 0 0 1 0
Delaware Northeast 0 0 1 0
Florida South 1 0 0 0
Georgia South 1 0 0 0
Hawaii West 0 1 0 0
… … … … … …
Dichotomous Variables
● When using dichotomous variables, one has to be left out of the model
● The dichotomous variable left out is called the reference level
Dichotomous Variables
7
● The p-value for each dichotomous variable tests for a significant difference between that category and the reference level
Dichotomous Variables
8
● Predicting market share with one categorical variable: region of the country
● If “midwest” is the reference level:
Dichotomous Variables
9
What is an interaction?
Two predictor variables interact when the effect on the response variable of one predictor variable depends on the value of the other.
Interaction/Moderation • X and M interact to predict Y
• X is the “independent variable”
• M is the “moderator” or “moderating variable”
• The effect of X on Y is moderated by M
• you have a theory...
11
• X and M interact to predict Y • X is the “independent variable” • M is the “moderator” or “moderating variable” • The effect of X on Y is moderated by M • you have a theory...
Interaction/Moderation
A Two-predictor Interaction Regression Function with Moderator M:
● β 0
= the expected response when X = 0 and M = 0
● But now, β 1
and β 2
can no longer be interpreted as the change in the mean response with a unit increase in the predictor variable, while the other predictor variable is held constant at a given value.
The Moderated Regression Equation
β 1
= “main effect” of X
The effect of X when M is zero.
β 2
= “main effect” of M
The effect of M when X is zero.
β 3
= interaction between X and M
For main effects to be meaningful, zero must be meaningful value.
Missing Data
● If a case has missing data for one of the explanatory variables, it will be left out of the regression
● If there is lots of missing data, your sample size could be greatly reduced
● Consider leaving out variables with lots of missing data, especially if your sample size is small to begin with
15
Causation
● A significant explanatory variable in a regression model indicates association, but not necessarily causation
CAUSALITY CAN ONLY BE INFERRED FROM A RANDOMIZED EXPERIMENT!!!!
16
Modeling an Interaction Term, with Statsmodels
In the marketing experiment here, suppose that Lobster Land wants to test out consumer reactions to four types of hats. Each hat has the same basic Lobster Land logo design, except one is purple, one is red, one is blue, and the other is white. To conduct the test, Lobster Land announced “Free Hat Day” on a Saturday. The park offered each of the first 1000 visitors a free hat, and allowed the visitors to select from among the four colors shown below:
A Marketing Experiment With Categorical Outcomes
In this experiment, we will assume that the null hypothesis is that there is no meaningful difference in consumer preferences -- in other words, park visitors as a whole are indifferent among the four options. Given this null hypothesis, our expected numbers of purple, dark red, blue, and white hats are 250, 250, 250, and 250, respectively. The observed values are the actual sums of hats of each color that were selected by the first 1000 visitors that day. The outcome values for purple, red, blue, and white were 315, 225, 240, and 220, respectively.
To analyze these results, we will use a table (see the following slide)
Color Observed Expected Observed-Expected (Observed-Expected)^2 (O-E)^2 / E
Purple 315 250
Red 225 250
Blue 240 250
White 220 250
The “expected” values shown here are the ones that we would see if consumers showed no preference for any particular type of hat. The observed values are the actual values that we saw when we ran this experiment at Lobster Land.
Our null hypothesis is that Purple = Red = Blue = White. The alternative hypothesis is that there is some difference between these options.
Color Observed Expected Observed-Expected (Observed-Expected)^2 (O-E)^2 / E
Purple 315 250 65 4225
Red 225 250 -25 625
Blue 240 250 -10 100
White 220 250 -30 900
Color Observed Expected Observed-Expected (Observed-Expected)^2 (O-E)^2 / E
Purple 315 250 65 4225 16.9
Red 225 250 -25 625 2.5
Blue 240 250 -10 100 0.4
White 220 250 -30 900 3.6
To get our chi-squared statistic, we must sum the values in the column on the right.
In a chi-square goodness of fit test, the degrees of freedom are found by taking the number of categories and subtracting 1. For this experiment, therefore, we have three degrees of freedom. With the chi-square value and the number of degrees of freedom in hand, we can find our p-value from a chi-square table -- these tables often published in the back of Statistics textbooks as part of the appendix. There are also many online tools that will generate a p-value for us after we provide it with a chi-square value and the number of degrees of freedom. The chi-square calculator on mathsisfun is one such example. (Please note the British usage of ‘maths’ in the plural).
With a p-value of .00003., we will reject the null hypothesis. We do this because our p-value here is suggesting a very tiny percentage chance (.003%) that we would see a chi-square statistic similar to or greater than the one we saw, if the null hypothesis were true.
import scipy, scipy.stats observed_values=scipy.array([315,225,240,220]) expected_values=scipy.array([250,250,250,250])
scipy.stats.chisquare(observed_values, f_exp=expected_values)
Experimental Design in the Digital Age
Experiment if you can: “No Causation without Experimentation”
Chapter 9
Experimentation – Definition
An experiment is a procedure in which one or more causal variables – for example, a marketing message about a product – are systematically manipulated and data on the effect variable – for example, choice to buy the product or not – are gathered while controlling for other variables that may influence the effect variable (e.g., Christmas season, weather, etc.)
26
Causal Inference
● Requirements for Causal Statement X → Y ○ (1) X must occur before Y ○ (2) There must be evidence of association between X and Y ○ (3) Control of other causal factors ○ (1) is typically manipulated. (2) can be assessed by the data.
(3) is typically done by randomizing, matching, or blocking ● Example:
○ product advertisement on TV → Choice of product purchased
27
Experiments – Procedure
• Decide on your treatments
• Design the experiment design
• Recruit participants and run the experiment
• Analyze the experiment
28
Experiments – Procedure
● Decide on your treatments ○ What are you trying to test? ○ How many factors? ○ What will you do if you have the information?
29
Design the Experiment
Potential experimental designs
• After only
• Multi-factor
• Before and after
• Between subject vs. within Subjects
Marketing Analytics Brennan Davis
30
Marketing Analytics Brennan Davis
31
Two groups, after only design:
G1(R) A O 1
G2(R) B O 2
• The estimate of the treatment effect X is: (O 2
– O 1
)
• Advantage is that there is no pre-test bias
• Assumes O 1
= O 2
• Randomization will help to insure that O 1
= O 2
but there are no guarantees
• Will work well with large sample sizes where randomization has a better chance of working
Pizza Study
• We will demonstrate aspects of experimentation using the Pizza Study
• The researchers ran experiments to explore a hypothesis that people tend to choose the same number of units of food regardless of its small or large size
• They called this the fixed unit effect
32
Defining The Fixed Unit Effect
● In the presence of others, people consume fixed number of food units based on what they perceive to be an appropriate quantity of units of a given food.
● The result ○ Unit size is held constant ○ Total caloric intake varies with unit size (i.e., smaller or larger
units), and consumers will consume more calories with larger unit sizes.
33
Pizza Study 1: Example of a 1 Factor Experiment
● Hypothesis 1: People will choose the same number of small (16ths) versus large slices (8ths), causing them to eat fewer overall calories.
● Design: 2(unit size: small vs. large) ● Procedure: Participants were randomly assigned to see one of the two
following pizzas and then decide how many slices they would take: ● Dependent variables:
○ Pizza Slices Selected ○ Calories Selected
34
Average Pizza Slices and Calories Selected
Pizza Study 1 Results
F(1, 58) = 21.86, p < .001
Hypothesis 1 is supported.
Multi-Factor
● The pizza tested a single factor: unit size. This factor had two levels: small and large
● However, it is possible to test multiple factors. ● For example, they next tested the pizza study as
○ Unit size: Small and large pizza slices ○ Presence of others: with others or alone
● Why would they do this? They hypothesize that the effect of unit size will differ when eating with others versus alone.
36
Pizza Study 2: Example of a 2 Factor Experiment
Hypothesis 2:
The presence of others will change the way unit size affects the number of units and total amount chosen:
● When eating with others, people will eat the same number of small (16ths) versus large slices (8ths), causing them to eat fewer overall calories.
● When eating alone, people will eat more small (16ths) versus large slices (8ths), causing them to eat the same number of overall calories.
37
Pizza Study 2: Example of a 2 Factor Experiment
Design: 2 (unit size: small or large) × 2 (presence of others: with others or alone)
● Procedure: Put people in a room with others or alone and gave them large whole pizzas cut into large or small slices
● Two assistants recorded number of units eaten
38
Study 2: Results (A) Units Chosen (B) Calories
Hypothesis 2 is supported.
Number of Factors Factors can be any number of things; these are typical:
•Product
•Promotion
•Place
•Price
•Segment
Text recommends 2 or 3 factor designs in which:
•1-2 factors are one of the 4 Ps (e.g., price, $10 versus $20)
•1 factor is a segmentation variable (e.g., high versus low income)
40
The Language of Experimental Variables
• Dependent variable – the outcome variable
• Independent variable – manipulated variable for which the marketing analyst is interested in seeing an effect on the dependent (outcome) variable
• Moderator variable – it changes the way the independent variable affects the dependent variable
41
Two Group Before-After Design
• G1(R) O 1
A O 2
• G2(R) O 3
B O 4
• The estimate of the treatment effect X is: (O 2
- O 1
) - (O
4 - O
3 )
• Dropouts (differ across the two groups?)
• Reactive bias (reaction to the pre-treatment measurement. Is it the same in both groups?)
42
Potential Experimental Design Within or Between Subjects Design?
● A within-subjects design is an experiment in which the same group of subjects serves in more than one treatment. Note that I'm using the word "treatment" to refer to levels of the independent variable, rather than "group".
● A between-subject design is an experiment that has two or more groups of subjects each being tested by a different testing factor simultaneously.
● Within subjects designs are advantageous because you get greater statistical power due to “internal matching” (you are your own control)
● However, in some cases, due to contamination, time constraints, and infeasibility, then “between subjects” designs must be used
43
Study 4: Within Subjects Example
Procedure
● There are two rooms: one in which people are alone and another in which they are with others. Whereas before, we randomly assigned people to be alone or with others, in this design, everyone goes to both rooms
● They are randomly assigned to see small or large cookies in both rooms.
● There are two chances to consume (two dependent variables) ● Do you see any issues with this design?
44
Study 4: Within Subjects
Participants
• Randomization • Sample size
• Recruitment
46
Randomization
• Randomization is typically the way that other variables are controlled for (e.g., flip a coin to decide what a subject receives)
• Randomization of subjects over treatments is the strategy for eliminating biases in measuring treatment effects due to selection between the experimental units.
• Helps control for other variables that may influence the effect and outcome variables
47
Recruitment
• Population
• Incentives
• Timing (all conditions at once)
• Potential pools
• What is a CONVENIENCE SAMPLE?
○ A convenience sample is drawn from that part of the population that is close at hand
• For the real world a convenience sample wouldn’t be good enough. Why not?
48
Validity
• Internal validity is the extent to which the observed results are due to the experimental manipulation
• External validity is the degree to which the experimental results are likely to hold beyond the experimental setting
• Usually there is a tradeoff between the two
• Without internal externality, external validity means nothing ==> Fix the level of internal validity and strive for the highest possible external validity.
49
• Suppose I need to know how many city residents are obese.
• Bathroom scale (suppose Mr. X weighs 196 lbs.)
– 180, 200, 240, 160, 200
– 220, 220, 220, 220, 220
– 196, 196, 196, 196, 196
● Validity: I am measuring accurately
● Reliability: I can depend on the measure
Validity and Reliability
50
1st measurement is neither valid nor reliable.
2nd measurement is reliable, but not valid
3rd measurement is reliable AND valid
Threats to Internal Validity
1. History effect: Events external to the experiment that affect the responses of the people involved in the experiment
2. Maturation effect: Changes in the respondents that are a consequence of time, such as aging, getting hungry, or getting tired.
3. Pre-test effect: The fact that someone has been measured previously might affect their future behavior
4. Instrument variation: The method used to collect data changes within the experiment (e.g., questionnaire, interviewer, etc.)
5. Selection effect: If units self-select themselves into the treatment and control groups then this is of serious concern if the selection reason is related to the outcome of interest.
6. Mortality (or drop out): The sample becomes unrepresentative
Threats to External Validity
• All 6 listed for internal validity could also impact external validity
• Non-representative sample, environment, and materials used
• Reaction bias
– the respondents you observe are distorted because the units behave differently knowing they are being observed
52
Laboratory Versus Field Experiment (and AB Test)
• Validity: A field experiment tends to have a higher external validity but a smaller internal validity
• Exposure: By doing a field study, you may provide information to competitors or adversely affect the marketplace
53
A/B Testing Everything you ever wanted to know about A/B testing.
Chapter 8
● Understand the principal concepts of an effective promotional campaign
● Explain the scientific method ● Define A/B testing ● Recognize potential tools for A/B testing ● Identify key issues and problems with A/B testing ● Know how to implement a simple A/B test
Learning Objectives
Last decade, Tropicana wanted a new juice box...
So, they redesigned the package.
New box launched in thousands of stores nationwide
How did it go?
• “Ugly, stupid, cheap and generic looking”
• Ad agency defends it
• Sales plunge 20%
• Agency CEO fired
• Total cost: $33 Million
8 Months + $33 Million Dollars = This Package Fails
There has to be a better way!
A/B Tests!
Data-Driven Processes
Listen to Your Customers
Follow the Scientific Method
The Scientific Method
Believe it or not, some people leave Facebook.
Let’s pretend we work at Facebook:
Ask Question How can we stop more people from deactivating their Facebook account?
Do research The reason why people use Facebook is because their friends are on Facebook.
Hypothesis If you remind people about their friends who are being left behind, we will lower the deactivation rate.
Facebook’s Actual A/B Test Control:
Experiment:
The guilt trip worked. Control:
Experiment:
Reduced Deactivation Rate by 7%✔
Guess the Winner
Quiz: Guess the Winner
A B
✔ 80% improvement on sign
ups Source: http://www.abtests.com/
Guess the Winner
Quiz: Guess the Winner A
B
+60% signups
Source: http://www.abtests.com/
✔
Designing a Test
• Anyone at the company should contribute
• Ideas can come from user testing (your regressions, for example)
• Use best practices
• Be creative, try crazy ideas
• Use common sense: don’t piss off your users for the sake of metrics
73
The 41 Shades of Blue Test at Google
Link Color
Marketing Analytics Brennan Davis
Quick Crash Course: Hexadecimal Color Values
Hexadecimal values are built with a base-16 numbering system. In base 16, the number values for 0 through 9 are just 0 through 9. A = 10, B = 11, C = 12, D = 13, E = 14, F= 15.
A hexadecimal color value with six digits is composed of a Red value (two digits), Green value (two digits), and a Blue value (two digits).
To find any particular color value, multiply the right-most digit for that color by 1, then multiply the leftmost digit for that color by 16. Add those two products. That’s it! The values go from 0 to 255. 256^3 = 16,777,216!
Why hex? We can contain as much information in six digits as binary would enable with 24 digits!
What color is Google blue? Red = 2 + 32 = 34. Green = 0, and Blue = 1 + 192 = 193. RGB = (34, 0, 193)
What color is Bing blue? Red = 0, Green = 4 + 64 = 68, and Blue = 12 + 192 = 204 RGB = (0, 68, 204)
Warning: The Local Maxima
Source: Joshua Porter
A/B Testing
A/B Testing = A/B/N Testing = Split Testing
AD654 Takeaway: The “N” in A/B/N refers to the number of variants being used in the test. An A/B test does not always have to involve only two options.
import pandas as pd abtest = pd.read_csv(“abtest.csv”) abtest.head()
The dataset includes a unique userID for each observation, along with that person’s group number, and the amount of money that the person spent during the concert.
Things get a little crazy in the park around LobsterPalooza Festival time!
from scipy import stats t, p = stats.ttest_ind(abtest.loc[abtest['group'] == 1, 'concert'].values,abtest.loc[abtest['group'] == 2, 'concert'].values, equal_var=False)
The p-value for our t-test is extremely low. At a significance level far greater than 99 percent, we can state that the variation in the concert spending between members of the two groups is not the result of random chance. We will reject the null hypothesis that there is no meaningful variation among the groups.
Other Key Points
To run a successful A/B test, it is essential that the users are randomly assigned to either group. If the split is created in any other way, the results of the test cannot be considered valid. Subjects in the test are not notified that there is some sort of a test going on.
Splitting your consumers based on the chronological order of their sign-up date, their total spending, or a demographic feature such as their age means that a confounding variable has now entered the experiment.
Since A/B testing is designed to identify cause-and-effect relationships, a confounding variable would throw the results of such a test into question.
Furthermore, valid interpretation of A/B test results depends on the key assumption that the only changed variable is the one that the experimenter wishes to explore.
This ensures that there is not some other confounding variable that might be influencing the results. In multivariate testing, which is separate from A/B testing, marketers perform experiments in which multiple variables are altered simultaneously.
A multivariate test of a website could involve a change in the navigation menu layout and a change in color scheme. With three different menu layouts and four different color schemes, there would be 12 total unique combinations to compare.
To the best degree possible, an A/B test should be conducted when there are as few outside variables as possible that could influence someone’s actions.
An A/B test of a feature on a job search site could be hard to interpret, as there are so many factors that could impact whether a job seeker takes a particular action.
Presumably, the user of such a site might abruptly stop after being hired, so any test whose results hinge upon some future action would be questionable. An online dating site, or a site that matches service providers with clients, could face a similar issue.
But Can It Be JUST ONE Thing?
That depends on what you’re trying to prove.
Let’s compare these two pictures (used as part of a class activity last semester).
How many differences can we identify?
We can use a statistical test to tell us “The concert picture works better than the beach picture” but we can’t be more specific than that. In other words, we can’t say anything here about “guitar vs. no guitar” or about “black shirt vs. white shirt,” etc.
Statistical Tests
These are not the only statistical tests that could be used for this analysis, BUT for AD654:
● Chi-square goodness of fit to compare categorical outcomes vs. expectations. ● Two-sample t-test
Two Sample t-test
If you have more than one outcome to test? You can still use this. Let’s say we have Marketing Campaign A, Marketing Campaign B, and Marketing Campaign C.
We can test A vs. B, A vs. C, and B vs. C.
However, there can be a reason why ANOVA is preferred in such scenarios -- to think about why, consider margin of error. If we run one test with a .05 margin of error, then we’re generally okay. But as we add more tests, we add more “error range.”
Chi-Square Goodness of Fit
