2 Assignments wk3 in 24 hours

Is Decision-Based Evidence Making Necessarily Bad?

S U M M E R 2 0 1 0 V O L . 5 1 N O. 4

R E P R I N T N U M B E R 5 1 4 1 9

Peter M. Tingling and Michael J. Brydon

SUMMER 2010 MIT SLOAN MANAGEMENT REVIEW 71COURTESY OF HERMAN MILLER

Some decisions can’t be made based on evidence at all. The facts are too ambiguous — such as the rejection by focus groups of the aesthetics of Herman Miller’s Aeron chair.

DECISION MAKING IS the essence of manage-

ment, which explains why so much attention

continues to be focused on how to do it better. In

recent years, much has been written about evidence-

based — or fact-based — decision making. The core

idea is that decisions supported by hard facts and

sound analysis are likely to be better than decisions

made on the basis of instinct, folklore or informal

anecdotal evidence. One need look no further than

the shelves of the local bookstore to see an unprece-

dented collection of well-written titles extolling the

virtues of data and analysis, such as Competing on

Analytics (Davenport and Harris), Moneyball (Lewis)

and Super Crunchers (Ayres). These books, like deci-

sion-making courses in business schools and the

prescriptions of management consultants, focus on

how to improve decision outcomes through improved

process and technique. Many organizations have

heeded the call and have invested heavily in data pro-

cessing infrastructure and analytic tools, based on the

assumption that better evidence-based decisions will

follow naturally from these investments. While this

focus on evidence is a welcome change from “thin slicing” or purely instinctive or

intuitive snap judgments, these prescriptions tend to downplay the more fundamental

questions: What is the relationship between evidence and the decision process that an

organization actually uses? Why is evidence collected in the first place?

Our research and consulting experience suggests that evidence is not as frequent an

Many managers think they’ve committed their organizations to evidence-based decision making — but have instead, without realizing it, committed to decision-based evidence making. Is that all bad? What can be done to fix it? BY PETER M. TINGLING AND MICHAEL J. BRYDON

Is Decision-Based Evidence Making Necessarily Bad?

THE LEADING QUESTION Managers want ‘fact- based’ decisions. Are they getting them?

FINDINGS ! Evidence is not as

frequent an input to decisions as suggested by the business press.

! Not all decisions use evidence in the same way. Evidence can be used to make, inform or sup- port a decision.

! Managers need to be aware that evidence is shaped by subordinates to meet perceived expectations of company leaders.

E S S AY

72 MIT SLOAN MANAGEMENT REVIEW SUMMER 2010 SLOANREVIEW.MIT.EDU

E S S AY

input to a decision process as suggested by the popu-

lar press. For example, we recently studied a major

North American financial institution as it consid-

ered a proposal to change its enterprise e-mail

platform from one technology to another. The orga-

nization had conducted two prior reviews of e-mail

systems from major vendors and had twice recom-

mended remaining with the existing supplier.

However, the head of a small but influential and

profitable division of the company advocated switch-

ing platforms in order to provide better integration

with a specialized tool used only within his division.

When asking staff to conduct the third major analy-

sis, a director of the company’s information

technology group recommended that the project

manager produce a report that would support a

change of vendor. A project team member told us,

“The executives have already made up their minds….

We are being told that this is the way that we are going,

we need to get on board — be team players — and

make the decision work out to be [the new choice].”

Clearly, the ideal evidence-based decision pro-

cess was subverted in this case by the perceived

requirement to marshal facts and analysis to sup-

port a decision that had already been made elsewhere

in the organization. We call this practice decision-

based evidence making and argue that it is more

widespread than many managers acknowledge. The

purpose of this article is to examine the practice and

to answer three fundamental questions:

■ Why does decision-based evidence making occur

in organizations?

■ Is decision-based evidence making necessarily bad?

■ And, if decision-based evidence making is inevi-

table in organizations, what can be done to lessen

its negative impacts?

Why Does Decision-Based Evidence Making Occur? Managers use different approaches when making

decisions, ranging from highly analytical and algo-

rithmic to ad hoc and intuitive. Rather than

converging on a single best approach, most norma-

tive models of decision making emphasize context

and adaptability. The models suggest that a deci-

sion-making approach should be tailored to fit the

particular characteristics of the decision problem.

Thus, an algorithmic approach is well suited to

highly structured decision problems in which the

ends and means are well understood. Intuition and

bargaining are more appropriate for poorly struc-

tured decision problems with multiple, conflicting

ends and uncertain means.

The problem with the flexible, contextual ap-

proach is that the role of evidence is unclear. In some

cases, hard evidence is critical in determining a deci-

sion outcome. In other cases, evidence is merely

symbolic; it is used to lend legitimacy to the decision

and signal the rationality of the decision makers. In

this article, we impose structure on this loose contin-

uum by identifying three distinct roles for evidence in

decision-making practice, depending on whether it is

being used to make, inform or support a decision. (See

“The Role of Evidence in Decision Making.”)

Make a Decision Evidence is used to make a decision whenever the

decision follows directly from the evidence. For

example, consider the choice of the optimal loca-

tion of a new distribution facility. The objective is

to minimize the cost of the facility, where cost is a

function of several quantifiable factors such as

route lengths, demand patterns, land availability

and local tax incentives. Qualitative and noneco-

nomic factors do not fit well into this mode of

decision making and must be either ignored or

transformed into quantitative evidence through

“pricing out” or similar techniques. The objective

facts regarding each of the decision alternatives (the

potential facility locations) are then used as inputs

into an optimization algorithm, and the location

with the overall minimum cost is provided as out-

put. The combination of data, a cost model and an

optimization algorithm make the decision with

minimal human intervention.

The success of evidence-based decision making

in highly structured environments such as location

planning and supply chain management is univer-

sally acknowledged. The recent push toward

evidence-based decision making in medicine sug-

gests that even incomplete or provisional evidence

(expressed as probabilities) can be valuable in less-

structured, ambiguous decision environments.

Indeed, while research shows that many managers

have yet to adopt analytic approaches, head-to-head

comparisons in which algorithmic, evidence-based

SLOANREVIEW.MIT.EDU SUMMER 2010 MIT SLOAN MANAGEMENT REVIEW 73

techniques are evaluated against impressionistic

and intuitive judgments of experts show that algo-

rithmic techniques often provide better results, even

in unstructured decision contexts such as granting

parole or predicting job satisfaction.

The primary risk of making decisions by relying

exclusively on hard evidence is that the algorithms and

models used to transform evidence into a decision

provide an incomplete or misleading representation of

reality. There are many examples in finance, for exam-

ple, where the models used by traders misspecified

important real-world dependencies and risks. As the

collapse of companies such as Long-Term Capital

Management LP and The Bear Stearns Companies Inc.

attests, the organizational downside created by a com-

mitment to misspecified models can be significant.

Inform a Decision Evidence is used to inform a decision whenever the

decision process combines hard, objective facts with

qualitative inputs, such as intuition or bargaining

with other stakeholders. The role of evidence in in-

forming decisions is thus akin to due diligence. For

example, in a succession planning decision, objective

evidence about the candidates’ past performance in

managerial roles is often an important input to the

decision process. However, subjective, impressionis-

tic information is typically combined with hard

evidence when making the final decision. Bargaining,

expressions of power and other organizational ele-

ments that do not fit within the orthodox model of

rational choice may also enter into such decisions.

The evidence-based inputs to the decision

process either confirm or disconfirm the decision

makers’ initial subjective beliefs and preferences. If

the evidence is confirmatory, decision makers can

move forward, confident that they have “the num-

bers” required to support their choice. However, a

dilemma arises if the evidence disconfirms the ini-

tial subjective decision. The decision makers must

either trust the evidence (in which case they have

implicitly switched to the make mode described

above) or side with their gut.

When asked, managers report that they routinely

grant evidence priority over their impressionistic

assessments. For example, in a survey of the capital

budgeting practices of large American companies,

45% of respondents said they would reject a capital

investment opportunity that had a favorable “stra-

tegic analysis” if the net present value (NPV) of the

opportunity was negative. However, as illustrated in

the example of the enterprise e-mail system, execu-

tives often provide analysts with subtle (and, in

some cases, unsubtle) signals regarding the desired

outcome of a formal, evidence-based analysis. Our

research shows that senior decision makers are

often unaware that evidence has been shaped by

subordinates to conform to the perceptions of

THE ROLE OF EVIDENCE IN DECISION MAKING Not all decisions incorporate evidence in the same way, or intend to marshal it toward the same end. This chart shows three roles that evidence can play, depending on whether the aim is to make, inform or support a decision.

ROLE OF EVIDENCE IN DECISION MAKING DESCRIPTION ARCHETYPAL DECISION RISKS

Make decision Evidence forms the basis of the decision

Facilities location Poor decisions due to misspecified models

Inform decision

Evidence is one of several inputs to the decision process

Diagnosis, strategic planning

Mismatch between evidence and other inputs requires shift to “make” or “support” roles

Support decision Evidence is created to support a decision made using other inputs

New product development, technology adoption

Demoralization of analysts; poor decisions due to decision biases and false consensus

Evidence DecisionDecision Process

Evidence

DecisionIntuition, Experience,

Bargaining, etc.

Decision Process

DecisionIntuition, Experience,

Bargaining, etc.

Decision Process

Evidence

SLOANREVIEW.MIT.EDU74 MIT SLOAN MANAGEMENT REVIEW SUMMER 2010

E S S AY

management. Top managers may receive little dis-

confirming evidence and, as a consequence, may

underestimate the extent to which evidence is being

trumped by intuition.

Support a Decision Evidence is used to support a decision whenever the

evidence is gathered or modified for the sole purpose

of lending legitimacy to a decision that has already

been made. This is distinct from making decisions

without formal evidence. Organizations are rou-

tinely faced with complex, poorly structured

decisions for which unambiguous evidence favoring

a course of action is simply unavailable. To cite a

classic example, Herman Miller Inc. discovered that

focus groups disliked the novel aesthetics of its new

Aeron chair. Management of the company ultimately

chose to disregard the disconfirming evidence pro-

vided by the focus groups and launched the new

chair anyway. The Aeron chair — which went on to

become enormously successful — is not an example

of decision-based evidence making because no evi-

dence was manufactured to support bringing the

chair to market. Instead, the culture of Herman

Miller was such that it was possible to make a deci-

sion without or in spite of empirical evidence.

In other organizations, formal evidence is held

in much higher regard and disconfirming evidence

cannot simply be dismissed. Much depends on the

cultural and formal norms of the organization and

its external stakeholders. Decision-based evidence

making is most prevalent whenever evidence is

highly valued within the organization (that is, evi-

dence is effectively “mandatory”) but a conflict

exists between the evidence and the decision mak-

ers’ strongly held beliefs. Decision makers in these

circumstances cannot simply override the discon-

firming evidence (as in the case of the Aeron chair)

but are instead more likely to rewrite the evidence

so that it supports their beliefs.

A natural consequence of the push toward evi-

dence-based decision making is that the norms

requiring evidence are becoming increasingly

explicit and rigid. For example, Harrah’s Entertain-

ment Inc. CEO Gary Loveman requires that the

effectiveness of new incentive programs at the com-

pany’s casinos be confirmed with small-scale

experiments before a decision can be made to imple-

ment the programs companywide. This is not to

suggest that the implicit pressure to manufacture er-

satz evidence is a new phenomenon. Indeed, during

Robert McNamara’s “whiz kids” era at the Ford Motor

Company, interns reportedly cut up copies of The

Wall Street Journal and inserted them into binders

and boxes. This voluminous “evidence” was wheeled

into the boardroom to demonstrate the depth of the

analysis supporting their recommendations.

Is Decision-Based Evidence Making Necessarily Bad? The downside of organizational contempt for dis-

confirming evidence is clear. Consider Vince

Kaminski’s frustrating tenure as the managing direc-

tor for research at Enron Corp. prior to the company’s

demise. Each deal undertaken by Enron was sup-

posed to have a complete risk analysis conducted by

Kaminski and Enron’s team of approximately 50

highly skilled mathematicians and analysts. How-

ever, as Enron became increasingly focused on deal

volume and increasingly hostile to facts and analysis,

the risk analysis became a charade.

The implosion of Enron has joined the Bay of Pigs

invasion and the space shuttle Challenger disaster in

the canon of cautionary tales warning decision mak-

ers of the perils of selective fact reading. But even if

decision-based evidence making does not result in a

high-profile fiasco, the practice sends powerful sig-

nals to others about organizational priorities, who

has power and who does not, and the value the orga-

nization places on facts and analysis. In short, the way

in which an organization’s senior leadership uses evi-

dence during decision making matters deeply to the

people responsible for creating the evidence.

We watched a few years ago as a large architec-

tural and engineering company decided to replace

its conventional computer monitors with smaller,

sleeker LCD monitors. Although such a decision

could have been justified in many different ways

(such as improved aesthetics, reduced power con-

sumption or simply employee satisfaction), the

company chose to construct an elaborate formal

cost-benefit analysis that included the fact that the

new monitors required less physical space. However,

as was pointed out by a senior manager involved in

the process, these benefits were imaginary: After all,

the firm would not actually recoup any of its real

SLOANREVIEW.MIT.EDU SUMMER 2010 MIT SLOAN MANAGEMENT REVIEW 75

estate costs following the purchase of the flat moni-

tors. Finally, after others began to question the

legitimacy of the analysis, the managing director

stopped the process and made the decision by fiat.

A habitual reliance on ceremonial decision pro-

cesses and devalued evidence can have serious

organizational side effects. Massive investments in

information systems and analytical tools are wasted

if the critical resources in the organization’s analytic

capability are repeatedly demoralized and humili-

ated by having their efforts dismissed, overruled or

altered. These analysts may begin to self-censure and,

anticipating management expectations, present only

confirming evidence. On the other hand, many deci-

sion environments exist where evidence (based on

historical data) cannot possibly tell the whole story.

Many important phenomena are inherently unpre-

dictable, and an analysis of historical evidence is

unlikely to provide much insight.

To illustrate, consider our experience with a mid-

sized credit union that was contemplating a switch

from its existing wealth management system to a

new one developed by a startup company. A formal

analysis of the two alternatives showed that they

were similar in terms of projected ROI and other

conventional financial measures. However, the head

of wealth management preferred the system from

the startup company because its modern architec-

ture offered the promise of increased flexibility. The

chief operating officer, in contrast, worried about

the survivability of a startup and the risk of being

stranded with an unsupported system.

Neither executive could find convincing support

for their positions. On one hand, the head of wealth

management did not know how his business would

evolve over the next few years and was therefore un-

able to estimate the “option value” of increased

flexibility offered by the new platform. On the other

hand, software is often subject to direct and indirect

increasing returns to adoption. As the COO pointed

out, their incumbent platform was widely used

within the industry. The stability provided by a large

installed base was extremely important to the credit

union — perhaps more important than the func-

tionality of the software itself. Markets for such

products tend to tip in favor of a single industry

standard. However, as owners of Betamax and HD

DVD players will attest, predicting the winner in

such a standards war is extremely difficult.

Clearly, the leadership of the credit union could

not rely on historical evidence and formal models;

instead, they had to place bets on the future of their

business based on intuition and rough consensus.

However, both executives recognized the signal their

actions would send to other organizations. Small,

regional credit unions tend not to compete directly

against each other and have a tradition of sharing

both infrastructure and knowledge. The head of

wealth management recognized that several other

smaller credit unions were facing the same decision

regarding their wealth management platforms and

would be more likely to follow the larger credit

union’s lead if they believed that the decision to

switch was the result of careful, rational economic

analysis. The executives reasoned that their decision

to switch to a new platform would be used as

evidence in the decision processes of other organi-

zations. By signaling rigor and rationality in their

own decision process, they could perhaps trigger an

information cascade or herding effect. Herding by

other companies would benefit the credit union by

giving the startup provider a large enough installed

base to obviate concerns about long-term surviv-

ability. The head of wealth management therefore

prepared a comprehensive formal business case to

support the credit union’s decision to switch plat-

forms. The report was ultimately shared with other

credit unions and, as predicted, several of these or-

ganizations decided to migrate to the new platform.

As this example illustrates, decision-based evi-

dence making is not always a practice to be avoided.

As the credit union executives recognized, evidence

can be used not only to make or inform a decision but

also to support a decision, to signal rationality and to

instill others with confidence that a good decision was

made. Whether such signals are effective and whether

they ultimately help or hurt the organization depends

critically on the nature of the decision itself and the

intended audience for the manufactured evidence. As

the example of the architectural and engineering

company illustrates, decision-based evidence making

that is directed at a well-informed internal audience is

almost always perceived negatively. It typically under-

mines the legitimacy of the decision it was intended

to support. In contrast, decision-based evidence

can be effective when the audience is external and

76 MIT SLOAN MANAGEMENT REVIEW SUMMER 2010

E S S AY

SLOANREVIEW.MIT.EDU

the manufactured evidence supports the organiza-

tion’s best guesses about a complex and unpredictable

decision environment.

What Can Be Done to Lessen the Negative Impact of Decision- Based Evidence Making? One way for organizations to avoid negative decision-

based evidence making is for decision makers to have

a clear understanding of the different roles evidence

can and should play in a decision process. This im-

plies that decision makers within the organization

have the flexibility to determine what constitutes le-

gitimate justification of a particular decision. The

difficulty that arises in practice is that organizations

may have trouble differentiating between situations

in which disconfirming evidence should be heeded

and situations in which disconfirming evidence can

safely be dismissed. Clearly, Enron was wrong to sys-

tematically ignore the formal analysis of its risk

management team. However, the Aeron chair was an

enormous commercial success, thereby vindicating

Herman Miller’s decision to discount the negative re-

action to the chair by focus groups.

Ultimately, the leadership of the organization

must take responsibility for such judgments. To help,

we provide the following guidelines:

1Understand the nature of the decision problem

and assess the potential contribution of formal

evidence to the quality of the decision process. There

are many different types of problems — ranging

from new product development to adoption of

emerging technology standards — in which evi-

dence based on historical data provides little insight.

Decision makers should have the courage and the

organizational support in such environments to

make and justify a decision based on intuition, ex-

perience and consultation with others.

2Weigh the risks, costs and benefits of evidence

when advocating an evidence-based approach to

decision making. The costs should include not only

the time required to collect evidence but any nega-

tive signals created by decision-based evidence

making. For example, the decision by the architec-

tural and engineering company to replace its

conventional monitors with flat monitors was minor

in comparison to the company’s capital and operat-

ing budgets. The incremental benefit of a formal

business case was outweighed by the losses in pres-

tige and legitimacy caused by management’s initial

insistence on bogus evidence.

3Differentiate between internal and external au-

diences when engaging in decision-based

evidence making. As noted above, there are situa-

tions in which evidence has significant ceremonial

and signaling value. However, internal stakeholders

(such as employees) typically have much better

access to information than those outside of the

organization. Consequently, internal audiences are

seldom fooled by decision-based evidence making.

4Ensure that the objective evidence painstakingly

gathered by your analysts is reflected more often

than not in the decisions of the organization. If you

must feed manufactured evidence to internal audi-

ences, do so only rarely and sparingly. Enron provides

an example of an organization in which a disregard

for evidence and analysis became endemic.

There is mounting evidence in favor of evi-

dence-based decision making in a wide range of

organizational decision environments. The resistance

of many managers to rational and analytical decision-

making techniques is thus surprising. But what is

troubling is that many managers who believe they

have committed their organizations to evidence-based

decision making (and have made hefty investments to

back up this commitment) have committed instead to

decision-based evidence making. Methodology alone

cannot and should not replace managerial discretion

or judgment. But, in much the same way that a street-

light can be used for illumination or support depend-

ing upon the need, greater understanding of the

multiplicity of ways that evidence is used within orga-

nizations can lead to better decision making.

Peter M. Tingling is founder and CEO of Octothorpe Software Corp. Michael J. Brydon is an associate professor of management at Simon Fraser University. Comment on this article or contact the authors at [email protected].

Reprint 51419.

Copyright © Massachusetts Institute of Technology, 2010.

All rights reserved.

PDFs ■ Permission to Copy ■ Back Issues ■ Reprints

Articles published in MIT Sloan Management Review are copyrighted by the Massachusetts Institute of Technology unless otherwise specified at the end of an article.

MIT Sloan Management Review articles, permissions, and back issues can be purchased on our Web site: www.pubservice.com/msstore or you may order through our Business Service Center (9 a.m.-7 p.m. ET) at the

phone numbers listed below. Paper reprints are available in quantities of 250 or more.

To reproduce or transmit one or more MIT Sloan Management Review articles by electronic or mechanical means (including photocopying or archiving in any information storage or retrieval system) requires written permission. To request permission, use our Web site (www.pubservice.com/msstore), call or e-mail: Toll-free: 800-876-5764 (US and Canada) International: 818-487-2064 Fax: 818-487-4550 E-mail: [email protected]

Posting of full-text SMR articles on publicly accessible Internet sites is prohibited. To obtain permission to post articles on secure and/or password-protected intranet sites, e-mail your request to [email protected]

Customer Service MIT Sloan Management Review PO Box 15955 North Hollywood, CA 91615

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.