Article Review | Subject: Business Statistics (see below)

www.hbr.org

How to Design Smart Business Experiments

by Thomas H. Davenport

Included with this full-text

Harvard Business Review

article:

Idea in Brief—the core idea

Idea in Practice—putting the idea to work

1

Article Summary

2

How to Design Smart Business Experiments

Managers now have the tools

to conduct small-scale tests

and gain real insight. But too

many “experiments” don’t

prove much of anything.

Reprint R0902E

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.

How to Design Smart Business Experiments

page 1

Idea in Brief Idea in Practice

C O

P Y

R IG

H T

© 2

0 0 9 H

A R

V A

R D

B U

S IN

E S

S S

C H

O O

L P

U B

L IS

H IN

G C

O R

P O

R A

T IO

N . A

L L

R IG

H T

S R

E S

E R

V E

D .

•

Too many business innovations are launched on a wing and a prayer—de- spite the fact that it’s now reasonable to expect truly valid tests.

•

With a small investment in training, readily available software, and the right encouragement, an organization can build a “test and learn” capability.

•

Companies that equip managers to per- form small-scale yet rigorous experi- ments don’t only save themselves from expensive mistakes—they also make it more likely that great ideas will see the light of day.

You or someone on your team is suggesting a change that just might work. But why act on a hunch when you can hold out for evidence? According to the author, the best way to support decision making on potential innovations is to...

•

Design an experiment.

Start with a hypothesis about how the change will help the business. If it’s a good one, you’ll learn as much by disproving it as you would by proving it. Put it to the test by measuring what happens in a test group versus a control group. From the outset, be clear on what you need to measure to produce a decisive re- sult—and whether that’s a metric you even have the capability to track.

•

Act on the facts.

Nothing but a success in a testing environ- ment should be rolled out more broadly. But neither should failures simply be scrapped. Refine the hypothesis on the basis of the re- sults, and consider testing a variation. Most important, capture what’s been learned, and make it available to others in the organization through a “learning library,” so resources aren’t wasted proving the same thing again.

Example:

Marketers at the Subway restaurant chain wanted to drum up business by putting foot-long subs on sale for only $5, but fran- chise owners worried that the promotion would lure existing customers away from higher-priced menu items. An experiment pitting test sites against control sites proved that the promotion would pay off— which it subsequently did.

•

Make testing the norm.

Create the training and infrastructure that will enable nonexperts in statistics to oversee rig- orous experiments. Off-the-shelf software can walk them through the steps and help them analyze results. A core group of experts can lend resources and expertise and maintain the learning library. Leadership must cultivate a test-and-learn culture, in part by penalizing those who act without sufficient evidence.

As your managers become more comfortable with testing, they’ll discover that it paves the way for, rather than throwing up barriers to, promising new ideas.

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.

How to Design Smart Business Experiments

by Thomas H. Davenport

harvard business review • february 2009 page 2

C O

P Y

R IG

H T

© 2

0 0 9 H

A R

V A

R D

B U

S IN

E S

S S

C H

O O

L P

U B

L IS

H IN

G C

O R

P O

R A

T IO

N . A

L L

R IG

H T

S R

E S

E R

V E

D .

Managers now have the tools to conduct small-scale tests and gain real

insight. But too many “experiments” don’t prove much of anything.

Every day, managers in your organization take steps to implement new ideas without having any real evidence to back them up. They fiddle with offerings, try out distribution ap- proaches, and alter how work gets done, usu- ally acting on little more than gut feel or seem- ing common sense—“I’ll bet this” or “I think that.” Even more disturbing, some wrap their decisions in the language of science, creating an illusion of evidence. Their so-called experi- ments aren’t worthy of the name, because they lack investigative rigor. It’s likely that the resulting guesses will be wrong and, worst of all, that very little will have been learned in the process.

Take the example of a major retail bank that set the goal of improving customer service. It embarked on a program hailed as scientific: Some branches were labeled “laboratories”; the new approaches being tried were known as “experiments.” Unfortunately, however, the methodology wasn’t as rigorous as the rhetoric implied. Eager to try out a variety of ideas, the bank changed many things at once in its “labs,”

making it difficult if not impossible to deter- mine what was really driving any improved re- sults. Branches undergoing interventions weren’t matched to control sites for the most part, so no one could say for sure that the out- comes noted wouldn’t have happened anyway. Anxious to head off criticism, managers did provide a control in one test, which was de- signed to see if placing video screens showing television news over waiting lines would shorten customers’ perceived waiting time. But rather than looking at control and test groups, they compared just one control site with one test site. That wasn’t enough to en- sure statistically valid results. Perceived wait- ing time did drop in the test branch, but it went up substantially in the control branch, despite no changes there. Those confounding data kept the test from being at all conclu- sive—but that’s not how the findings were pre- sented to top management.

It doesn’t have to be this way. Thanks to new, broadly available software and given some straightforward investments to build ca-

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.

How to Design Smart Business Experiments

harvard business review • february 2009 page 3

pabilities, managers can now base consequen- tial decisions on scientifically valid experi- ments. Of course, the scientific method is not new, nor is its application in business. The R&D centers of firms ranging from biscuit bak- ers to drug makers have always relied on it, as have direct-mail marketers tracking response rates to different permutations of their pitches. To apply it outside such settings, however, has until recently been a major undertaking. Any foray into the randomized testing of manage- ment ideas—that is, the random assignment of subjects to test and control groups—meant employing or engaging a PhD in statistics or perhaps a “design of experiments” expert (sometimes seen in advanced TQM programs). Now, a quantitatively trained MBA can over- see the process, assisted by software that will help determine what kind of samples are nec- essary, which sites to use for testing and con- trols, and whether any changes resulting from experiments are statistically significant.

Consumer-facing companies rich in transac- tion data are already routinely testing innova- tions well outside the realm of product R&D. They include banks such as PNC, Toronto- Dominion, and Wells Fargo; retailers such as CKE Restaurants, Famous Footwear, Food Lion, Sears, and Subway; and online firms such as Amazon, eBay, and Google. As randomized testing becomes standard procedure in certain settings—website analysis, for instance—firms build the capabilities to apply it in other cir- cumstances as well. (See the sidebar “Stop Wondering” for a sampling of tests conducted recently.) To be sure, there remain many busi- ness situations where it is not easy or practical to structure a scientifically valid experiment. But while the “test and learn” approach might not always be appropriate (no management method is), it will doubtless gain ground over time. Will it do so in your organization? If it’s like many companies I have studied, an invest- ment in software and training will yield quick returns of the low-hanging-fruit variety. The real payoff, however, will happen when the or- ganization as a whole shifts to a test-and-learn mind-set.

When Testing Makes Sense

Formalized testing can provide a level of un- derstanding about what really works that puts more intuitive approaches to shame. In the- ory, it makes sense for any part of the business

in which variation can lead to differential re- sults. In practice, however, there are times when a test is impossible or unnecessary. Some new offerings simply can’t be tested on a small scale. When Best Buy, for example, ex- plored partnering with Paul McCartney on an exclusively marketed CD and a sponsored con- cert tour, neither component of the promo- tion could be tested on a small scale, so the company’s managers went with their intu- ition. At Toronto-Dominion, one of the largest and most profitable banks in Canada, testing is so well established that occasionally managers are reminded that, in the interests of speed, they can make the call without a test when they have a great deal of experience in the rel- evant business domain.

Generally speaking, the triumphs of testing occur in strategy execution, not strategy for- mulation. Whether in marketing, store or branch location analysis, or website design, the most reliable insights relate to the potential impact and value of tactical changes: a new store format, for example, or marketing pro- motion or service process. Scientific method is not well suited to assessing a major change in business models, a large merger or acquisition, or some other game-changing decision.

Capital One’s experience hints at the natural limits of experimental testing in a business. The company has been one of the world’s most ag- gressive testers since 1988, when its CEO and cofounder, Rich Fairbank, joined its predeces- sor firm, Signet Bank. You could even say the firm was founded on the concept. One thing that appealed to Fairbank about the credit card industry was its “ability to turn a business into a scientific laboratory where every decision about product design, marketing, channels of communication, credit lines, customer selec- tion, collection policies and cross-selling deci- sions could be subjected to systematic testing using thousands of experiments.”

1

Capital One adopted what Fairbank calls an information- based strategy, and it paid off: The company be- came the fifth-largest provider of credit cards in the United States.

Yet when it came time to make the largest decision the company had faced in recent years, Capital One’s management concluded that testing would not be useful. Realizing that the business would need other sources of capi- tal to remain independent, the team consid- ered acquiring some regional banks in order to

Thomas H. Davenport

(tdavenport@ babson.edu) is the President’s Distin- guished Professor of Information Tech- nology and Management at Babson College in Babson Park, Massachusetts. His newest book is

Competing on Ana- lytics: The New Science of Winning

, with Jeanne G. Harris (Harvard Business Press, 2007).

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.

How to Design Smart Business Experiments

harvard business review • february 2009 page 4

The real payoff will

happen when the

organization as a whole

shifts to a test-and-learn

mind-set.

transform itself from a monoline credit pro- vider into a full-service bank. The decision was not tested for a couple of important reasons. First, the nature of the opportunity made it im- perative to move quickly; no time was avail- able for even a small-scale test. Second, and more critical, it was impossible to design an ex- periment that could reliably predict the out- comes of such a major change in business di- rection. Still, after making the acquisitions, Capital One reaffirmed its commitment to information-based strategy. Its managers im- mediately set about translating that ethos into the full-service banking context, which re- quired pushing the method further, into tests involving customer service and employee be- havior. As one employee told me, “It’s much easier to do randomized testing with direct- mail envelopes than with branch bankers.”

Sears Holdings provides another example of what can reasonably be tested and what can’t. Interestingly, this is another business with a heritage of testing. Robert E. Wood, who origi- nally moved Sears out of the catalog business and into retail stores, said his favorite book was the

Statistical Abstract of the United States

. When he opened Sears’s first free-standing re- tail stores, in 1928, he placed two in Chicago. Asked why he needed two in one city, Wood said it was to reduce the risk of choosing a wrong location or store manager.

Today Sears Holdings has embarked upon a new era: Its primary owner, financier Edward Lampert, who has been its chairman since Kmart acquired Sears, is exploring alternative ways to combine the two troubled chains. To my knowledge, Lampert didn’t test the idea of combining the retailers. That would have been difficult if not impossible to do (and the jury is still out on whether the acquisition was a good decision). However, he’s a strong advocate of testing at the tactical level. He wrote in a 2006 letter to shareholders, “One of the great advan- tages of having approximately 2,300 large- format stores at Sears Holdings is that we can test concepts in a few stores before undertak- ing the risk and capital associated with rolling out the concept to a larger number of stores or to the entire chain.” The retailer has tested, for example, various formats for including Sears merchandise in Kmart stores, and vice versa, as well as other formats, such as the arrangement of merchandise in Sears stores by rooms in a consumer’s home (kitchen, laundry room, bed-

room, and so on). Beyond using the tactical-versus-strategic cri-

terion, there are other ways to decide whether formal testing makes sense. For instance, it is useful only in situations where desired out- comes are defined and measurable. A new sales training program might be proposed, but before you can test its efficacy, you’ll need to identify a goal (such as “We want to increase cross-selling”), and you must be able to mea- sure that change (do you even track cross- selling?). Sales and conversion-rate changes are frequently used as dependent variables in tests and are reliably measured for separate pur- poses. Other outcomes, such as customer satis- faction and employee engagement, may re- quire more effort and invasiveness to measure.

Tests are most reliable where many roughly equivalent settings can be observed. This might mean physical sites, as with Sears’s stores, or it might mean more ephemeral settings, such as alternative website versions. Among the earli- est and most extensive users of testing are retail and restaurant chains. Because so much is held constant among their multitudinous sites, it is easy to designate which ones will serve as ex- periments and which will serve as controls and to attribute cause to effect. By the same token, workplace design changes are most readily tested in companies that have offices in many cities. Drawing statistical inferences from small numbers of test sites is much more difficult and represents the leading edge of the test-and- learn approach.

Finally, formal testing makes sense only if a logical hypothesis has been formulated about how a proposed intervention will affect a busi- ness. Although it’s possible to just make a change and then sit back and observe what happens, that process will inevitably lead to a hypothesis—and often the realization that it could have been formulated in advance and tested more precisely.

The Process of Testing

To begin incorporating more scientific man- agement into your business, you’ll need to ac- quaint managers at all levels with your organi- zation’s process of testing. It is probably simple to grasp (a typical depiction is shown in the exhibit “Put Your Ideas to the Test”), but it must be communicated in the same terms to people across the organization. Having a shared understanding of what constitutes a

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.

How to Design Smart Business Experiments

harvard business review • february 2009 page 5

valid test enables the innovators to deliver on it and the senior executives to demand it.

The process always begins with the creation of a testable hypothesis. (It should be possible to pass or fail the test based on the measured goals of the hypothesis.) Then the details of the test are designed, which means identifying sites or units to be tested, selecting the control groups, and defining the test and control situa- tions. After the test is carried out for the speci- fied period—which sometimes can take several months but is usually done in less time—the data are analyzed to determine the results and appropriate actions. The results are ideally put into some sort of “learning library” (although, unfortunately, many organizations skip this step). They might lead to a wider rollout of the experiment or further testing of a revised hypothesis.

More broadly, managers must understand how the testing process fits in with other busi- ness processes. They conduct tests in the context of, for example, order management, or site se- lection, or website development, and the testing feeds into various subprocesses. At CKE Restau- rants, which includes the Hardee’s and Carl’s Jr. quick-service restaurant chains, the process for new product introduction calls for rigorous test- ing at a certain stage. It starts with brainstorm- ing, in which several cross-functional groups de- velop a variety of new product ideas. Only some

of them make it past the next phase, judgmen- tal screening, during which a group of market- ing, product development, and operations peo- ple will evaluate ideas based on experience and intuition. Those that make the cut are actually developed and then tested in stores, with well- defined measures and control groups. At that point, executives decide whether to roll out a product systemwide, modify it for retesting, or kill the whole idea.

CKE has attained an enviable hit rate in new product introductions—about one in four new products is successful, versus one in 50 or 60 for consumer products—and executives say that their rigorous testing process is part of the reason why. If you have had occasion to enjoy a Monster Thickburger at Hardee’s, or a Philly Cheesesteak Burger or a Pastrami Burger at Carl’s Jr., you’ve been the beneficiary of CKE’s efforts. These are just three of the successful new products that were rolled out after testing proved they would sell well.

At eBay, there is an overarching process for making website changes, and randomized test- ing is a key component. Like other online busi- nesses, eBay benefits greatly from the fact that it is relatively easy to perform randomized tests of website variations. Its managers have conducted thousands of experiments with dif- ferent aspects of its website, and because the site garners over a billion page views per day, they are able to conduct multiple experiments concurrently and not run out of treatment and control groups. Simple A/B experiments (com- paring two versions of a website) can be struc- tured within a few days, and they typically last at least a week so that they cover full auction periods for selected items. Larger, multivariate experiments may run for more than a month.

Online testing at eBay follows a well-defined process that consists of the following steps:

• Hypothesis development • Design of the experiment: determining

test samples, experimental treatments, and other factors

• Setup of the experiment: assessing costs, determining how to prototype, ensuring fit with the site’s performance (for example, mak- ing sure the testing doesn’t slow down user re- sponse time)

• Launch of the experiment: figuring out how long to run it, serving the treatment to users

• Tracking and monitoring

Stop Wondering

Testing is used to make tactical decisions in a range of business settings, from banks to retailers to dot-coms. Here are some questions various companies are examining:

•

Do lobster tanks increase lobster sales at Food Lion supermarkets?

•

Does a Kmart with a Sears store inside sell more than an all-Kmart format?

•

Do eBay users bid higher in auc- tions when they can pay by credit card?

•

What’s the optimum number of loose checks for a Wells Fargo ATM to accept?

•

Do Subway promotions on low-fat

sandwiches increase sandwich sales?

•

Does a Famous Footwear store sell fewer shoes when there is a compet- itor in the same mall?

•

Does a Toronto-Dominion branch get significantly more deposits when open 60 hours a week com- pared with 40?

•

Which promotional offers will most efficiently drive checking account acquisition at PNC Bank?

As a result of their testing, these organizations are finding out whether supposedly better ways of doing business are actually better. Once they learn from their tests, they can spread confirmed better practices throughout their business.

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.

How to Design Smart Business Experiments

harvard business review • february 2009 page 6

• Analysis and results The company has also built its own applica-

tion, called the eBay Experimentation Plat- form, to lead testers through the process and keep track of what’s being tested at what times on what pages.

As with CKE’s new product introductions, however, this online testing is only part of the overall change process for eBay’s website. Ex- tensive offline testing also takes place, includ-

ing lab studies, home visits, participatory de- sign sessions, focus groups, and trade-off analysis of website features—all with custom- ers. The company also conducts quantitative visual-design research and eye-tracking studies as well as diary studies to see how users feel about potential changes. No significant change to the website is made without extensive study and testing. This meticulous process is clearly one reason why eBay is able to introduce most

Put Your Ideas to the Test

1: Create or Refine Hypothesis

Ascertain

that the hypothesized relationships haven’t already been tested and measured— and that they can be.

Make

sure the hypothesis could generate substantial economic value.

Determine

whether it suggests an actual de- cision or action. (If not, go no further.)

2: Design Test

Ensure

that the number of test and control sites is sufficient for statistical significance.

Use

simulation to explore multiple strategies for creating control groups (for instance, they may be nearly identical but different on one key variable).

Assess

whether control group strategies pre- viously used for similar tests will suffice; they usually do.

Conduct

statistical analysis to minimize the number of test cells needed.

Extend

testing period if key metrics are highly variable.

3: Execute Test

Meet

with test and control site managers and analytical experts to discuss what might go wrong and what would constitute test- confounding events.

Instruct

field personnel to report abnormal events.

Remove

sites from test if test-confounding events occur.

Adjust

evaluation and compensation plans for managers so that they are not negatively affected by tests.

4: Analyze Test

Ensure

that “lift” from interventions is statis- tically significant.

Use

software to analyze results and manage

complex data from multiple test and control sites.

Determine

need for further testing.

Examine

as many site attributes as possible to see how key variables interact.

5: Plan Rollout

Study

attributes of test sites to determine whether rollout should be universal or differ- entiated.

Balance

complexity of rollout with ease of implementation and management.

6: Rollout

Stagger

the rollout and view it as a test in it-

self. (Are early-adopting sites yielding the de- sired result? If not, modify the approach in later-adopting sites.)

Encourage

site managers to share rollout strategies and tactics.

Learning Library

Develop

a summary of each test: hypotheses, test dimensions, key results, interactions, and rollout strategies and results.

Employ

standard business taxonomy to allow easy searching of library.

Make

library widely accessible to employees; publicize tests and results of important stud- ies to encourage a test-and-learn culture.

ANALYZE TEST

CREATE OR REFINE

HYPOTHESIS

DESIGN TEST

ROLLOUT PLAN

ROLLOUT

EXECUTE TEST

Adapted from Applied Predictive Technologies’ “Test and Learn” Wheel

LEARNING LIBRARY

2 1

456

3

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.

How to Design Smart Business Experiments

harvard business review • february 2009 page 7

changes with no backlash from its potentially fractious seller community. The online retailer now averages more than 113 million items for sale in more than 50,000 categories at any given time.

EBay performed extensive online and offline testing, for example, in 2007 and 2008, when it changed its page for viewing items on sale. The page had not been redesigned since 2003, and both customers and eBay designers felt it lacked organization, had inadequate photo- graphs of items, and suffered from haphazard item placement and redundant functionality. After going through all the testing steps, eBay adopted a new site design. It posted photos 200% larger than those in the previous design, added a countdown timer for auctions with 24 hours or less to go, made more prominent the item condition and return policy, and included tabs to make shipping and payment fields eas- ier to navigate. It also included new security features to prevent unauthorized changes in site content. Each new feature and function was tested independently with control pages. Measures of page views and bid counts suggest that the redesign was very successful.

Building a Testing Capability

Establishing a standard process is the first step toward building an organizational test-and- learn capability, but it isn’t sufficient unto itself. Companies that want testing to be a reliable, ef- fective element of their decision making need to create an infrastructure to make that hap- pen. They need training programs to hone com- petencies, software to structure and analyze the tests, a means of capturing learning, a pro- cess for deciding when to repeat tests, and a central organization to provide expert support for all the above.

Managerial training.

At the very least, managers should learn what constitutes a ran- domized test and when to employ it. Capital One, for example, offers a professional educa- tion program on testing and experiment de- sign through its internal training function known as Capital One University. One benefit of hosting a program like this, rather than sending managers outside for training, is the greater emphasis on how the testing connects to upstream and downstream activities in the business.

Test-and-learn software.

Some firms, such as Capital One and eBay, have built their own

software for managing experiments, but sev- eral off-the-shelf options exist—the most com- mon ones being broad statistical packages and analytical tools like SAS. With every passing year, these tools make it more possible for nu- merate—but not statistically expert—users to conduct truly defensible experiments. Ease of design and analysis has been a particular focus at Applied Predictive Technologies, whose product leads users through the test-and-learn process, keeps track of test and control groups, and provides a repository for findings to be usefully accessed in the future.

Some software tools are tailored to particu- lar problems or industries. Several packaged tools, for example, are available for the analy- sis of manufacturing-quality experiments. Likewise, highly specialized tools exist for on- line-usage testing, such as the web analytics software sold by Omniture and WebTrends and the free tools provided by Google Analyt- ics. As of yet, unfortunately, no single soft- ware tool can help organizations with all test- ing types and contexts.

Learning capture.

If a firm does a substan- tial amount of testing, it will generate a sub- stantial amount of learning about what works and what doesn’t. Ideally employees through- out the company would share that knowledge and use it to guide future initiatives. But that happens at few organizations. The head of testing at one online firm admitted, “All of that knowledge is in my head, and we’d be in tough shape if I were hit by a bus.” One bank execu- tive justified a lack of shared learning, com- menting, “We should probably do more, but we’ve found that people need to learn from doing the test themselves, even if we’ve done it before many times.” People do learn through personal experience, but one would hope that it’s not the only possible way.

Some organizations, however, have begun to address the issue. Capital One captures the learning from its thousands of tests in an on- line knowledge management system and has experimented with an even more ambitious system that would use such learning to guide product managers as they develop new offer- ings. Famous Footwear takes a “billboard” ap- proach; for each test, it captures the results in a one-page document, circulates that through- out the organization, and posts it on the wall outside the testing office.

Regular revisiting.

One tricky aspect of es-

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.

How to Design Smart Business Experiments

harvard business review • february 2009 page 8

tablishing a long-term testing approach is de- termining when to retest. There is no way to know for sure when a test has become obso- lete; an experienced analyst needs to assess whether enough factors have changed in the environment to make previous results suspect. Famous Footwear executives feel that the re- tail store location context—their primary ap- plication area for testing—changes enough to merit retesting after about a year. Netflix con- cluded in 2006 that its five-year-old customer tests needed to be redone; the user base had evolved in that time from internet pioneers to mainstream society members. CKE Restau- rants has difficulty deciding whether to retest pricing, particularly in times when commod- ity prices are increasing fast. Ironically, it is human intuition, not testing or analytics, that must be applied to determine the need for re- testing.

Core resource group.

Most of the firms that do extensive testing have established a small, somewhat centralized organization to super- vise it. The group either actually does the test- ing, as at PNC Bank, Subway, and Famous Footwear, or—if testing is employed through- out the organization—serves as a resource for methodological and statistical questions, as at Capital One. At PNC Bank, the test-and-learn group (part of the bank’s knowledge manage- ment function, which reports to Marketing) views the promotion of its own services around the bank as a priority. It tries to build relationships and trust with key executives so that no major initiatives are undertaken with- out testing. Without a central coordination point, testing methods may not be sufficiently rigorous, and test and control groups across multiple experiments may confound one an- other. That said, it’s not always easy to influ- ence or coordinate testing even when a central group exists.

Creating a Testing Mind-Set

In addition to making the requisite changes in process, technology, and infrastructure, organi- zations also need to establish a testing culture. Testing costs money (though not as much as widespread rollouts of new tactics that don’t work), and it takes time. Senior managers have to become accustomed to, and even passionate about, the idea that no major change in tactics should be adopted without being tested by peo- ple who understand testing.

Ask for evidence.

CEOs who firmly believe in testing can change their entire organiza- tion’s perspective on the issue. When people claim that testing has confirmed the wisdom of their idea, have them walk you through the process they used, and demand at least the level of rigor outlined in the exhibit “Put Your Ideas to the Test.”

Give it teeth.

Gary Loveman at Harrah’s En- tertainment has said that “not using a control group” is sufficient rationale for termination at the company. Jeff Bezos of Amazon reportedly fired a group of web designers for changing the website without testing. Toronto-Dominion has a culture in which managers insist on tests for every major initiative involving customers or branches. The CEO, Ed Clark, is a PhD econ- omist who once noted that although the bank might not be perfect, “nobody ever criticizes us for not running the numbers.”

Sponsor tests yourself.

The best manage- ment teams in this regard have institutional- ized the process of doing and reviewing tests. At Famous Footwear, Joe Wood and his senior management team meet with the testing head every two weeks to discuss past tests, upcom- ing tests, and preliminary and final results. Wood says that the company has made testing a part of management’s dialogue and the orga- nization’s culture.

• • •

Testing may not be appropriate for every busi- ness initiative, but it works for most tactical endeavors. And it just isn’t that difficult any- more. It needs to come out of the laboratory and into the boardroom. The key challenges are no longer technological or analytical; they have more to do with simply making manag- ers familiar with the concepts and the process. Testing, and learning from testing, should be- come central to any organization’s decision making. The principles of the scientific method work as well in business as in any other sector of life. It’s time to replace “I’ll bet” with “I know.”

1. “Capital One Financial Corporation,” HBS case no. 9-700- 124.

Reprint R0902E

To order, call 800-988-0886 or 617-783-7500 or go to www.hbr.org

For the exclusive use of L. WATSON, 2019.

This document is authorized for use only by LANA WATSON in 2019.