question answer

Optimal_Database_Marketing_Strategy_Development_an..._----_11_-_Gains_Charts_and_Expected_Profit_Calculations.pdf

Home >Business & Finance homework help >Marketing homework help >question answer

Gains Charts and Expected

Profit Calculations

217

11 With the first Inside Source response model built, Keri Lee was concerned with how well the model would work in rollout. Would it predict respond- ers, as it suggests?

To answer Keri’s question, her analyst created what is called a gains chart (or lift chart) using the validation sample. She was told by her analyst that such a gains chart will yield a simulation of what she can most likely expect to occur in rollout regarding the model’s predictive strength. Once the pre- dictive strength of the model was confirmed, Keri was ready to select the most profitable names for promotion.

Her goal for the overall campaign was to generate at least 5% profit after overhead. Given such a goal, she determined that she must receive at least a 2.5% response rate. Using the results of the validation gains chart (with predictions), she selected all names meeting her criteria.

It has now been 12 weeks since the promotional mailing occurred. The mailing was slightly under forecast. Keri was anxious to reconcile the fore- casted gains to determine if the model was the cause. Unfortunately, Keri forgot to promote a sample of names not meeting her criteria. As a result, Keri will not be able to determine if the reason the mailing is under fore- cast is due to the regression model not holding up as forecasted or due to the change in promotional format.

As mentioned in Chapter 10, regression models can be built to assist thedirect marketer in predicting which customers on a database are most likely to order, pay, renew, and so on for a particular product offering. Once the final regression model is built for the specified marketing cam- paign, the next step is the actual application of the model to determine

Drozdenko11 2/26/02 6:17 PM Page 217

Drozdenko, Ronald G., and Perry D. Drake. Optimal Database Marketing : Strategy, Development, and Data Mining, SAGE Publications, 2002. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/nyulibrary-ebooks/detail.action?docID=996727. Created from nyulibrary-ebooks on 2020-07-20 13:41:01.

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

218 OPTIMAL DATABASE MARKETING

which names to promote and which names not to promote from the entire database for an upcoming marketing campaign.

You, as a direct marketer, have several options for selecting names via regression models, some of which are easier to implement than others, depending on the number of models being applied and the flexibility of your computer systems. This book considers two applications:

1. Selecting names based on a single regression “response” gains chart

2. Selecting names based on “expected profit” calculations, which incorporates multiple regression models and marketing cost and profit figures

This chapter reviews the steps required to implement both methods, with detailed examples of the applications in the field of direct marketing, including the development of the marketing forecast.

The Response Gains Chart ________________________________

After building a single regression model to help predict the customers most likely to respond to a particular product offer, you are ready to create an analysis and validation response gains chart to determine (a) model stability and (b) the most profitable customers on the database to promote.

An incremental response gains chart will provide you with a ranking of customers from those most likely to respond to those least likely to respond. On the basis of your marketing cost and profit figures for the product pro- motion under consideration, determine how many names you can promote.

To develop an incremental response gains chart on the analysis portion of the sample, follow these steps:

1. Score the analysis sample using the regression model. This scoring process is done in the same manner as the scoring example in Chapter 10 (see Exhibit 10.5). That is, each name in the analysis sample is passed through the model and a single score assigned to each name, indicating their overall relative likelihood of ordering.

2. Once names are scored, rank the names from highest scoring to lowest scoring.

3. Cut the ranked scores into 10 equal “buckets,” each representing 10% of the sample. For example, Bucket 1 will contain the top scoring 10% of your sample, Bucket 2 will contain the next highest scoring 10% of your sample, and so on. NOTE: In some cases, you may have tied scores at a bucket cutoff level. If this occurs, move your cutpoint for that bucket up (or down) until the score changes. As a result, some buckets may have slightly more (or less) than the desired 10%.

Drozdenko11 2/26/02 6:17 PM Page 218

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

Gains Charts and Expected Profit Calculations 219

4. For each bucket, calculate the order rate and the gain. The gain is simply the index value minus 100.

To develop an incremental gains chart on the validation portion of the sample, follow these steps:

1. Score the validation (or holdout) sample using the regression model.

2. Once names are scored, rank the names from highest scoring to lowest scoring.

3. Define the 10 buckets as determined on the analysis sample. For example, Bucket 1 on the analysis sample was defined as names having a score greater than or equal to 0.2139. Therefore, the names in Bucket 1 are defined as those names having a score greater than or equal to 0.2139 also (even if that represents more or less than 10% of the validation sample).

4. For each bucket, calculate the order rate and the gain.

ACME Direct test promoted a new product to 20,000 names from the customer segment “5� paid orders in the past 24 months” (see Exhibit 11.1).

The sample was split 10,000 for analysis and 10,000 for validation. At the request of the product manager, the analyst built a regression model (using the analysis sample only) predicting those customers who were most

Entire Customer File Universe Size = 4,670,513

Customers with past purchases that have been

written off Universe Size = 323,156

Remaining customers (the balance)

Universe Size = 3,813,391

Customers who frequently return merchandise (and

have no written-off purchases)

Universe Size = 533,966

Customers with 5+ paid orders in the past 24

months Universe Size = 1,256,479

Remaining Customers (the balance)

Universe Size = 2,556,912

20,000 sample taken from this customer

segment.

Exhibit 11.1 Segment Test Promoted a New Product Offering by ACME Direct

Drozdenko11 2/26/02 6:17 PM Page 219

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

220 OPTIMAL DATABASE MARKETING

likely to order the product of concern. Once the model was complete, the analyst scored both the analysis and validation samples with the final regression model, ranked the names from highest scoring to lowest scoring, and developed the incremental response gains charts (see Exhibit 11.2) following the steps previously outlined.

Tie scores must have been present, because many of the 10 buckets pro- duced on the analysis sample do not contain exactly 10% of the names.

The gain values displayed in the incremental gains charts are merely a function of the index values. They are calculated as the index to total for each bucket minus 100. Like the index values displayed for univariate tabulations (see Chapter 7), they allow you to more easily view how much better or worse each group or bucket of names performed when compared to the total and in relation to one another.

For the validation sample, the first bucket (those with a score greater than or equal to 0.7546) has a response rate 76% higher than the entire sample. In other words, this regression model has identified roughly 10% of this universe that will respond at a rate 76% higher than the response rate of the entire segment (7.48% vs. 4.25%).

Also notice that both the analysis and validation incremental gains charts have perfect monotonically (perfectly smooth) decreasing response rates when viewed from top to bottom. This is one sign of a good model.

If the incremental gains chart produced on the analysis sample reveals close to monotonically decreasing gains but it does not for the validation sample, it is an indication of a poor model. We advise you to examine the model for weak predictors with high p values or problems with multi- collinearity (see Chapter 10).

Exhibit 11.2 Incremental Response Gains Charts Built on the Analysis and Validation Samples

Incremental Gains on Analysis Incremental Gains on Validation

Gain Gain Score Sample Sample Response Over Sample Sample Response Over

Bucket Level Percent Count Rate Total Percent Count Rate Total

1 GE 0.7546 9.98 998 7.69 81 10.00 1,000 7.48 76 2 0.6532–0.7545 10.02 1,002 6.59 55 10.00 1,000 6.38 50 3 0.5589–0.6531 10.00 1,000 5.61 32 9.94 994 5.53 30 4 0.5013–0.5588 10.05 1,005 5.19 22 10.06 1,006 5.14 21 5 0.4429–0.5012 9.95 995 4.76 12 9.95 995 4.72 11 6 0.4013–0.4428 9.96 996 4.34 2 10.05 1,005 4.42 4 7 0.3521–04012 10.04 1,004 3.02 �29 10.04 1,004 3.19 �25 8 0.2782–0.3520 10.00 1,000 2.30 �46 9.96 996 2.38 �44 9 0.1891–0.2781 10.00 1,000 1.87 �56 9.98 998 2.00 �53 10 LE 0.1890 10.00 1,000 1.15 �73 10.02 1,002 1.28 �70 Total 100.00 10,000 4.25 0 100.00 10,000 4.25 0

Drozdenko11 2/26/02 6:17 PM Page 220

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

If the incremental gains charts produced on both the analysis and valid- ation samples are not close to monotonically decreasing and are very choppy, it may indicate that the model was not properly coded by your pro- grammer, and therefore, the samples were not properly scored.

As mentioned in Chapter 6, the purpose of a validation sample is to allow the analyst to confirm the analysis results. Remember that a sample is just that, a sample, and as such, a model built on any sample will have error associated with it. How much error your final model may have depends on how careful you were in the exclusion of insignificant or correlated variables. Assessing the model on the validation sample com- pared to the analysis sample allows you to gauge the amount of error associated with your model. If much error is present, we say the model was “overfit.”

How can we tell if the model built reveals an overfitting situation? If the projected gains for the top few buckets are similar between the analysis and validation samples, you have built a stable model. If this is the case, it implies you built little error into the model and it validates nicely on the holdout or validation sample. If the gains are significantly lower for the validation sample when compared to the analysis sample for the top few buckets, then you have a weak model that will not hold up in rollout.

For the gains shown in Exhibit 11.2, the top bucket in the analysis segment responded at a rate 81% higher than the entire sample, whereas the same bucket for the validation sample yielded a response rate 76% higher than the entire sample. This is a 6% ([81�76]/81 � 6%) falloff in gains. Anything greater than a 10% falloff in the top few buckets is cause for concern. A greater than 10% falloff indicates a potential problem with your regression model, and in such cases, we strongly advise you to carefully review your regression model for potential problems (multi- collinearity, weak predictors, etc.).

The only purpose of the incremental gains chart on the analysis sample is to check for falloff in gains when compared to the same on the validation sample. Once you are satisfied that the falloff in gains from analysis to valid- ation is minimal and acceptable, you will base all marketing decisions on the incremental response gains chart produced on the validation sample only. The validation sample provides you with a more stable indicator of what you can expect the model to yield in a rollout situation.

On the basis of the validation gains chart, what percentage of the names can the product manager profitably promote, given the fact that she needs at least a 2.97% response rate to break even? According to the validation gains chart, it appears that by promoting all customers with a regression score above 0.3521 (the top seven buckets), the breakeven criteria will be met. The product manager will not promote the last three buckets of names, because the projected response rates for each are below the required breakeven response rate level.

Gains Charts and Expected Profit Calculations 221

Drozdenko11 2/26/02 6:17 PM Page 221

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

222 OPTIMAL DATABASE MARKETING

Now that the product manager knows which buckets she will promote, she needs to determine the response rate for these seven buckets combined. To determine this, she will have to create a cumulative gains chart on the validation sample. Cumulative figures tell the associated number of names, the response rate, and the gain for all names falling into a given bucket and all buckets above it. These figures are determined from the incremental gains chart.

It is easy to turn any incremental gains chart into a cumulative gains chart using Excel. The cumulative gains chart shown in Exhibit 11.3 was created in Excel using the incremental gains chart figures created on the validation sample shown in Exhibit 11.2.

For example, the cumulative “Sample Count” for Bucket 2 represents the incremental “Sample Count” for Buckets 1 and 2 combined (1,000 � 1,000). The cumulative “Response Rate” for Bucket 2 represents the incremental “Response Rate” for Buckets 1 and 2 combined: [(1,000 � 0.0748) � (1,000 � 0.0638)]/2,000. The cumulative “Gain” for Bucket 2 is calculated as [(6.93/4.25) � 100] � 100. This information tells the product manager what she can expect if she promotes the top two buckets. Similarly, the cumulative figures for Bucket 3 represent the total names falling into Buckets 1, 2, and 3, the combined response rate for Buckets 1, 2, and 3, and the gains associated with the combined response rate for Buckets 1, 2, and 3. This information tells the product manager what she can expect if she promotes the top three buckets.

The product manager instructs her programmer to score all 1,256,479 customers falling into the “5� paid orders in the past 24 months” segment (see Exhibit 11.1) using the regression model built and to transmit only those

Exhibit 11.3 Cumulative Gains on Validation

Incremental Gains on Validation Cumulative Gains on Validation

Gain Gain Score Sample Sample Response Over Sample Sample Response Over

Bucket Level Percent Count Rate (%) Total Percent Count Rate (%) Total

1 GE 0.7546 10.00 1,000 7.48 76 10.00 1,000 7.48 76 2 0.6532–0.7545 10.00 1,000 6.38 50 20.00 2,000 6.93 63 3 0.5589–0.6531 9.94 994 5.53 30 29.94 2,994 6.46 52 4 0.5013–0.5588 10.06 1,006 5.14 21 40.00 4,000 6.13 44 5 0.4429–0.5012 9.95 995 4.72 11 49.95 4,995 5.85 38 6 0.4013–0.4428 10.05 1,005 4.42 4 60.00 6,000 5.61 32 7 0.3521–04012 10.04 1,004 3.19 –25 70.04 7,004 5.26 24 8 0.2782–0.3520 9.96 996 2.38 –44 80.00 8,000 4.90 15 9 0.1891–0.2781 9.98 998 2.00 –53 89.98 8,998 4.58 8 10 LE 0.1890 10.02 1,002 1.28 –70 100.00 10,000 4.25 0 Total 100.00 10,000 4.25 0 100.00 10,000 4.25 0

Drozdenko11 2/26/02 6:17 PM Page 222

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

names with a score greater than or equal to 0.3521 to the letter shop for pro- motion. How many names will the programmer be expected to transmit?

If the sample the regression equation was built on is truly a random and representative sample of all names falling into the “5� paid orders in the past 24 months” segment, the programmer should be expected to transmit exactly 70.04% of this universe, or 880,038 names. If the programmer transmits anything significantly higher or lower than this figure, a problem exists such as

♦ The database was not properly scored on the regression model, or ♦ A major shift in the composition of the universe being regressed must

have occurred since the time of the test promotion.

If 880,038 names are transmitted and thus promoted, how many orders should the product manager expect to receive from this campaign? She should expect to receive 46,290 orders (880,038 � 0.0526).

When creating a gains chart, there is nothing magical about 10 buckets. You can have as many or as few buckets as you wish; however, keep in mind that you do not want too few names falling into any one bucket or your estimated response rates for each bucket will have a high error associated with them. In addition, if your final regression model has few variables, you may not have the spread in scores to create 10 buckets and might need to produce your gains chart with fewer buckets. This is typically a problem when regressing “data poor” lists.

Gains Charts and Expected Profit Calculations 223

_________________ Options When Lacking Validation Samples

The importance of a validation sample cannot be stressed enough, espe- cially when an analyst has limited experience in modeling a particular product, list, or offer. If your budget does not allow for test samples large enough to be split into one for analysis and another for validation, two options are available to you.

Historical Gains Falloff Chart

If you have implemented response models in the past and can acquire some historical figures regarding forecasted and actual gains from these past campaigns, you can develop what is called a historical gains falloff chart. These are created by comparing, for several past campaigns, actual to fore- casted gains and determining the falloff observed. Once determined, you can apply this adjustment to future gains charts prior to rollout to ensure that the best possible decision regarding who to promote is made.

Drozdenko11 2/26/02 6:17 PM Page 223

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

224 OPTIMAL DATABASE MARKETING

For example, last year the product manager at a small newsletter pub- lishing house with two finance titles built a response model to determine which inactive customers she would promote for Hot Stock Tips. Unfortunately, not enough names were tested for the creation of a valid- ation sample. Therefore, the decision to determine who to promote was based on the same sample the model was built on. The product manager, on the basis of an analysis of profit and cost figures, determined she could only promote the top 30% (top three buckets) of the incremental gains chart. For these top three buckets, the cumulative gains chart revealed a gain of 75% in response over promoting everyone. In rollout, she received a gain of only 65%. This was a falloff in gains of 13%. The product manager’s boss was not happy. In all likelihood, if the product manager had had a validation sample available, the gain based on the validation sample would have been closer to what she actually obtained.

Next year, the product manager built a response model to determine which inactive customers she would promote for the other title, The Mutual Fund Report. Again, she lacked a large enough test sample for the creation of a validation sample. Therefore, the decision to determine who to promote was based on the same sample the model was built on. The product manager, on the basis of an analysis of profit and cost figures associated with this title, determined she could only promote the top 30% (top three buckets) of the incremental gains chart. For these top three buckets, the cumulative gains chart revealed a gain of 104% in response over promoting everyone. Is this what she used as her fore- casted gain? Absolutely not. She took advantage of her experience from last year’s campaign and decided to reduce the forecasted gain by 13% (the same loss in gain realized when promoting the same percentage of the file for Hot Stock Tips last year). Instead, the product manager will only forecast a gain of 90.5% (87% of the analysis gain). This is basic- ally how it works. When you lack samples large enough for the creation of a validation sample, you will adjust your gain based on historical information.

Ideally, this product manager will want to determine the historical loss in gains observed for all buckets, not just for Bucket 3 of a cumulative gains chart. And she will do so for several past campaigns. This way, if the prod- uct manager decides to promote, for example, the top six buckets for her next campaign, she will know how much to adjust the gains to ensure a more accurate forecast. In other words, she will want to construct a historical gains falloff chart. Exhibit 11.4 shows forecasted cumulative gains and actual cumulative gains received for two different book titles promoted last year by a small direct marketer of children’s books. In both cases, the forecasted gains were based on the same sample that the model was built on.

Drozdenko11 2/26/02 6:17 PM Page 224

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

Gains Charts and Expected Profit Calculations 225

Using the calculated falloff in cumulative gains observed from forecast to actual for both products, an average was determined. Using this average falloff in cumulative gains, the product manager is now in a position to know exactly how to adjust each bucket of a cumulative gains chart prior to finalizing future marketing forecasts.

This technique is by no means perfect, but it does give valuable infor- mation to help small direct marketers make better promotional decisions and build more stable forecasts. Falloff in gains is typically a function of an analyst’s experience and that experience in relation to the product, list, and offer being promoted. A more seasoned analyst typically builds models with less falloff than a new analyst.

Bootstrapping

Bootstrapping or bagging is another method that can be employed when samples are too small to be split into one for analysis and another for valid- ation. It is a bit more complex to implement than the gains falloff chart but will yield forecasted gains much closer to actual.

To bootstrap, many subsamples are taken from the main test sample and a regression model built on each. For example, a major entertainment com- pany test promoted a new music club offer to 10,000 names. Rather than build the response model on all 10,000 names, they will instead build many regressions on subsamples of this main sample. They will randomly take 100 subsamples from the main sample of size 1,000 each. For each of these 100 samples, they will build a regression model. Obviously, they will do so using a more automated technique such as the stepwise routine discussed in Chapter 10.

Exhibit 11.4 Assessment of Forecasted Versus Actual Gains

Book Title X Book Title Y Average

Bucket Score (%) Fcst. Actl. Falloff (%) Fcst. Actl. Falloff (%) Falloff (%)

1 10 125 106 –15.00 101 87 –14.00 –14.50 2 20 101 89 –12.00 90 78 –13.00 –12.50 3 30 90 82 –9.00 84 76 –10.00 –9.50 4 40 67 62 –8.00 74 67 –9.00 –8.50 5 50 42 40 –5.00 55 51 –7.00 –6.00 6 60 30 29 –4.00 40 38 –5.00 –4.50 7 70 22 22 –2.00 32 31 –3.00 –2.50 8 80 14 14 –1.00 20 20 –1.10 –1.05 9 90 5 5 0.00 10 10 –1.00 –0.50

10 100 0 0 — 0 0 — —

Drozdenko11 2/26/02 6:17 PM Page 225

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

Once all models are built, they will examine them and determine which variables consistently entered the models. For the final model, they will choose the variables that consistently entered each model some percentage (e.g., 75%) of the time. By employing this technique, they have avoided the use of any variable that may have been unstable in rollout.

In fact, many direct marketers are closely examining this technique as a replacement for the current analysis and validation methodology. The belief is that the result of a bootstrapped model is a more stable and robust model than a model built on analysis samples and validated on holdout samples.

For more information on bootstrapping, read the paper “On Bagging and Nonlinear Estimation” (May 1999) by Jerome H. Friedman, Stanford University, and Peter Hall, Australian National University (www-stat.stanford.edu/~jhf).

226 OPTIMAL DATABASE MARKETING

Expected Profit Calculations ______________________________

In business decisions, you may not only want to consider the likelihood of an event occurring but also consider the costs and profit figures associated with each of the events. Considering both enables you to base your mar- keting decisions on the likely payoff.

For example, you are promoting a magazine subscription offer for which the cancel rate is high. In determining who to promote, you build both a response model and a payment model to help you select not only customers most likely to order but customers most likely to order and pay. As previ- ously discussed, this is a case in which you will not be able to use a standard response gains chart. This is where the calculation of expected profit comes into play. To determine the expected profit or loss for a particular business scenario, multiply the probabilities associated with each possible outcome by their respective net costs or profit values and then sum.

The expected monetary value calculation (EMV) is written as

EMV � P(O1)M1 � P(O2)M2 � P(O3)M3 � . . . � P(On)Mn

Where P(O1), P(O2), P(O3), . . . P(On) � the probabilities associated with each of

the n possible outcomes of the business scenario and the sum of these probabil- ities must equal 1

M1, M2, M3, . . . Mn � the net monetary values (costs or profit values) associated with each of the n pos- sible outcomes of the business scenario

The easiest way to understand EMV is to review a lottery example. Assume you are considering the purchase of a lottery ticket in which the probability of winning the $1 million jackpot is 1 in 10 million. If

Drozdenko11 2/26/02 6:17 PM Page 226

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

the lottery ticket costs $1 to purchase, what is the expected monetary value or expected profit/loss for this scenario (purchasing of a lottery ticket)?

First, list the various outcomes (O1, O2, O3, . . . On) of this scenario. In this case, there are only two (n � 2):

Possible Outcomes

O1 � You win the lottery O2 � You don’t win

Next, determine the probabilities P(O1) and P(O2) associated with each of the two outcomes:

Possible Outcomes Probability

O1 � You win the lottery P(O1) � 1/10,000,000 � 0.0000001 O2 � You don’t win P(O2) � 9,999,999/10,000,000 � 0.9999999

Now, determine the associated net monetary values M1 and M2 for each of the two outcomes:

Possible Outcomes Probability Monetary Value

O1 � You win the lottery P(O1) � 1/10,000,000 M1 � $999,999 � 0.0000001

O2 � You don’t win P(O2) � 9,999,999/ M2 � �$1 10,000,000 � 0.9999999

Note: The net monetary value associated with winning the lottery is calcu- lated by taking the winnings of $1,000,000 and subtracting the $1 cost of the ticket. The net value is $999,999.

Finally, calculate the EMV as

EMV � P(O1)M1 � P(O2)M2 (probability you win)(value if you win) � (probability you don’t win)(value if you don’t win)

� (0.0000001)($999,999) � (0.9999999)( � $1) � $0.0999999 � $0.9999999 � � $0.90

In other words, if you play this particular lottery game, your expected payout is a loss of $0.90. Keep in mind that this is an average value, meaning that if you play this particular game over and over again, your average payout per play is a loss of $0.90.

To implement EMV calculations into your direct marketing campaigns, the regression models built predicting a customer’s likelihood of ordering, paying, renewing, and so on must be of the logistic form. As you recall from Chapter 10, a logistic regression model yields a probability

Gains Charts and Expected Profit Calculations 227

Drozdenko11 2/26/02 6:17 PM Page 227

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

associated with the action being modeled. A standard multiple regression model will not give you a probability. It should be noted, however, that there are ways to convert a multiple regression score into a probability based on gains chart information; however, this will not be covered in this book.

For the remainder of this section, assume that all regression models are of the logistic variety, meaning the scores assigned to each name (when passed through the model) are a probability bound between 0 and 1, thereby allowing use of the EMV formula.

A review of the important customer actions you wish to consider in the EMV formula is first required. Suppose you are interested in determining who to promote and not promote for a new product offer by assessing the probabilities and costs associated with a customer, taking each path shown in Exhibit 11.5.

To calculate the EMV for each customer, we must first write out the possible customer paths or outcomes:

♦ Order and pay ♦ Order and not pay ♦ Not order

Next, determine the probability of each path or outcome:

Outcome Probability of Outcome

Order and pay (Probability of ordering and paying) � (Probability of ordering) � (Probability of paying)

Order and not pay (Probability of ordering and not paying) � (Probability of ordering) � (Probability of not paying)

Not order (Probability of not ordering)

228 OPTIMAL DATABASE MARKETING

Promote Customer

Order the product Silent -

don’t order the product

Pay for the product Don’t pay for the product

Exhibit 11.5 List of Customer Actions of Concern When Promoted

Drozdenko11 2/26/02 6:17 PM Page 228

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

Note: The derivation for the probability formulas listed above will not be discussed here. For this application, it is only necessary for you to realize that the probability of “Event A and Event B both occurring” is equal to the probability of “Event A occurring” multiplied by the probability of “Event B occurring.” These rules follow the basic rules of probability and can be found in any intermediate level statistics book.

As you can see, we will need to build two logistic regression models in order to determine these probabilities: one predicting the probability of ordering and another predicting the probability of paying. To determine the probability that a customer will not order, you take 1 minus the prob- ability that a customer will order. To determine the probability that a cus- tomer will pay, you take 1 minus the probability that a customer will not pay.

Now determine the cost and profit figures associated with each of these three paths or outcomes.

Outcome Monetary Value of Outcome

Order and pay � Cost of the promotion piece � net value of a paid product (including average billing costs)

Order and not pay � Cost of the promotion piece � cost of an unpaid order (including average billing costs)

Not Order � Cost of the promotion piece

To determine most of these costs, your marketing finance department should be able to assist you. The cost of an unpaid order for a magazine subscription will include the cost of the copies already sent before stopping the subscription (called grace copies). Typically, two to three grace copies are sent. Regarding the sale of merchandise, the cost of an unpaid order includes costs associated with returns and kept merchandise (if applicable). When determining the cost of a return, keep in mind that a certain portion of the returned merchandise may be refurbishable and a certain portion not refurbishable.

Determining billing costs (if applicable) are fairly straightforward. For a paid order, use the average number of bills sent to get payment, multiplied by the average cost per bill. For an unpaid order, use the cost of the entire billing series for nonpayers and the average billing costs expended for those who return merchandise.

Calculate the EMV by multiplying the probabilities associated with each of the three actions previously listed by their respective cost and profit values:

EMV � [(Probability of ordering) � (Probability of paying) �(Value of a paid order)] � [(Probability of ordering) �(Probability of not paying) � (Value of an unpaid order)] �[(Probability of not ordering) � (Value of a nonorder)]

Gains Charts and Expected Profit Calculations 229

Drozdenko11 2/26/02 6:17 PM Page 229

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

230 OPTIMAL DATABASE MARKETING

Assume the following marketing costs and profit values associated with an upcoming promotional campaign:

Promotional cost per piece mailed: $0.87

Net value of a paid order (including average billing costs): $13.56

Cost of an unpaid order (including average billing costs): $6.56

Also assume the following probabilities of ordering and paying for two customers (Mr. Jones and Ms. Smith), scored on two logistic regression models:

Customer Probability of Ordering Probability of Not Paying

Mr. Jones 28% 5% Ms. Smith 34% 18%

Which customer is more profitable if promoted for this particular product?

EMV (Mr. Jones) � [(0.28) � (1 � 0.05) � ($13.56 � $0.87)] � [(0.28) � (0.05) � ( � $6.56 � $0.87)] � [(1 � 0.28) � (�$0.87)]

� $3.38 � $0.10 � $0.63 � $2.65

EMV (Ms. Smith)� [(0.34) � (1 � 0.18) � ($13.56 � $0.87)] � [(0.34) � (0.18) � (�$6.56 � $0.87)] � [(1 � 0.34) � (�$0.87)]

� $3.54 � $0.45 � $0.57 � $2.52

Although more likely to order (34% vs. 28%), Ms. Smith is less likely to pay (82% vs. 95%), resulting in an expected profit value less than that of Mr. Jones. Therefore, it is more profitable to promote Mr. Jones than Ms. Smith. If the promotion decision was based on a single regression response gains chart, Ms. Smith would have been erroneously promoted.

We used a very simple EMV exercise to present this topic so it would be easy to understand. In reality, these EMV models are much more complex. In the publishing industry, for example, the EMV formula may include 2-year values and consider the likelihood of renewing. For continuity clubs, the EMV formula may include the likelihood of future shipments being accepted and paid for by the customer.

Once you have calculated the expected profit values for each name in your customer segment, rank the names based on their EMV score from highest to lowest scoring, just like that for a response gains chart. Once done, a promotional decision will be made.

Drozdenko11 2/26/02 6:17 PM Page 230

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

Gains Charts and Expected Profit Calculations 231

Determining who to promote and not promote will be based on the EMV scores themselves. If the goal of your particular promotion is to break even, then you will select any name with an EMV score greater than or equal to $0.00.

Implementing EMV is also very useful when you have multiple offers and need to decide what offer to send a customer. For example, to deter- mine which of two product promotions to send to each customer on the database, you need to develop two models: one predicting a customer’s likelihood of ordering Product 1 and another predicting a customer’s like- lihood of ordering Product 2. Then score each customer on both equations and calculate their respective EMV scores. Promote each customer with the product for which they have the highest EMV.

_______________________________________ Reconciling Gains The model had been applied and the highest scoring names selected to be promoted. When the marketing campaign results are final and all orders have been received, the next step is to determine how well the model held up in rollout. That is, if you expected a 20% gain in response over pro- moting everyone, did you achieve that gain in rollout?

To reconcile the selection gains, a sample of names not meeting your criteria must be promoted. This is called a fails sample. For example, you promoted the top 50% of the names in your gains chart or the top five buckets. You received a 4% response rate. Unless you promoted a sampling of the bottom five buckets (the “fails”), you will have no way to determine the actual gain received in rollout over mailing everyone.

You do not need to promote a large fails sample; 5,000 or 10,000 names will suffice. Of course, the larger the fails sample, the better the estimate will be regarding how these fails would have actually performed in rollout had they all been promoted.

The importance of taking a fails sample when employing any selection criteria (manual selects or regressed) cannot be stressed enough. Without such a sample, you will not know if you were successful in defining your target market. In addition, if the results of a campaign were underforecast, you will not be able to isolate the problem (e.g., was it underforecast due to the regression gain or the new creative/format gain not holding up?).

For example, a cataloger test promotes a new children’s toy catalog featuring toys from around the world and receives a response rate of 2.54%. Due to the success of this unique children’s catalog, the analyst was asked by the product manager to build a response model to assist her in determining who to promote and not promote in rollout. On the basis of the incremental gains chart produced on the validation sample, the product

Drozdenko11 2/26/02 6:17 PM Page 231

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

232 OPTIMAL DATABASE MARKETING

manager decides to promote the highest scoring 40%, or top four buckets, of her primary customer segment. In addition, the product manager test promoted a sampling of 10,000 fail names (names falling into the bottom six buckets of her gains chart).

Assume the forecasted gain in response for these names (based on the cumulative gains chart) was �49%. Also assume that her primary customer segment was comprised of 4,352,145 names in total. Exhibit 11.6 is a report generated by this cataloger’s fulfillment shop showing the rollout results of this campaign 8 weeks after the mail date. Did the cataloger receive the forecasted gain of �49%?

The easiest way to determine if forecasted gains were obtained is through the creation of a reconciliation table, as shown in Exhibit 11.7. This table lists all known information regarding the pass (promoted) and fail (not-promoted) names for the universe being reconciled.

Because you did not promote all 4,352,145 names, you will estimate the response rate of the total universe by weighing together the actual pass response rate and the fails sample response rate. The weights used are equal to what the pass and fail names represent out of the total universe. In our example, the pass names represented 40% and the fails (in total) represented 60%.

Calculate the estimated response rate of the total universe as a weighted average of the pass and fail response rates:

Total Universe Response Rate � (3.80% � 0.40) � (1.77% � 0.60) � 1.52 � 1.06 � 2.58

Exhibit 11.6 Actual Campaign Results

Campaign Order Intake Report

Source # Promoted Orders Response Rate

Primary Universe: Top Scoring 40% 1,740,858 66,153 3.80% Fails Sample 10,000 177 1.77%

Exhibit 11.7 Reconciliation Table

List Size Percentage Response Rate (%) Gain

Pass Names 1,740,858 40% 3.80% ? Fail Names 2,611,287 60% 1.77%* Total Universe 4,352,145 100% ?

*Based on a sample of 10,000 names promoted.

Drozdenko11 2/26/02 6:17 PM Page 232

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

Now, given our estimated total response rate, we can calculate the gain achieved via the regression selection:

Estimated Gain in Response � [(3.80/2.58) � 100)] � 100 �� 47.3%

Actual was very close to forecast. In fact, we were only off by 3.4%— (49 � 47.3)/49—in forecasted gain.

Had the gains been off significantly (e.g., greater than 10%), we advise you to closely examine the regression model to determine which variable(s) did not perform as expected.

Gains Charts and Expected Profit Calculations 233

_______________________________________ Chapter Summary A response gains chart is used to determine who to promote and not pro- mote for an upcoming offer. A response gains chart reveals those customers most likely to order by ranking them based on their regression scores. Those with the highest regression score are the most likely to respond, and those with the lowest regression score are the least likely to respond. In addition, the gains chart is used to determine the stability of a regression response model. Gains on the analysis and validation samples are compared to assess model stability. If a major falloff in gains is observed between the analysis and validation samples, model stability is in question. In addition to response gains charts, names can also be ranked on expected profit values. Such values are a function of not only customers’ likelihood to order but also their likelihood to pay and marketing cost and profit figures.

_______________________________________ Review Questions 1. What are the steps used to create an incremental gains chart on the

analysis sample?

2. What are the steps used to create an incremental gains chart on the validation sample?

3. Why is it important to base the promotional decision on the gains chart produced on the validation sample rather than the analysis sample?

4. What options are available for forecasting gain based on a response model when lacking a validation sample?

5. What is the benefit of basing the promotional decision on expected profit as opposed to a simple response gains chart?

6. Why is it important to promote a sample of names not meeting the selection criteria when rolling out with a new selection criteria?

Drozdenko11 2/26/02 6:17 PM Page 233

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed

Drozdenko11 2/26/02 6:17 PM Page 234

C op

yr ig

ht ©

2 00

2. S

A G

E P

ub lic

at io

ns . A

ll rig

ht s

re se

rv ed