Essay for organizational behavior class

sirleo

reading4.pdf

Home >Psychology homework help >Essay for organizational behavior class

www.hbr.org

HBR

ARGE

Selection Bias and the Perils of Benchmarking

by Jerker Denrell

•

Impressive studies show that following best practices, focusing on the core, and building a strong culture are among the secrets of business success. But beware: These and other received ideas may also be the secrets of failure.

Reprint R0504H

For the exclusive use of S. Sadeghi, 2020.

This document is authorized for use only by Sohail Sadeghi in FE202 (A) Organizational Leadership taught by Koppman, S., University of California - Irvine from Sep 2020 to Mar 2021.

http://www.hbr.org

http://harvardbusinessonline.hbsp.harvard.edu/relay.jhtml?name=itemdetail&referral=4320&id=R0504H

HBR

ARGE

Selection Bias and the Perils of Benchmarking

by Jerker Denrell

harvard business review • april 2005 page 1

C O

P YR

IG H

T ©

2 00

5 H

A R

V A

R D

B U

SI N

E SS

S C

H O

O L

P U

B LI

SH IN

G C

O R

P O

R A

T IO

N . A

LL R

IG H

T S

R E

SE R

V E

D .

Managers learn by example. They—and the consultants they pay for advice—study the methods and tactics of successful companies in search of the magic formulas for business prosperity. What could make more sense?

What could be more dangerous. Looking at successful firms can be remarkably mislead- ing. I once listened to a presentation about the attributes of top entrepreneurs. Drawing on a wealth of impressive case studies, the speaker concluded that all of these leaders shared two key traits, which accounted for their success: They persisted, often despite initial failures, and they were able to persuade others to join them.

That sounded reasonable enough to most people in the audience. The only trouble was, the speaker failed to point out that these self- same traits are necessarily the hallmark of spectacularly unsuccessful entrepreneurs. Think about it: Incurring large losses requires both persistence in the face of failure and the ability to persuade others to pour their money down the drain.

Here’s the problem about learning by good example: Anyone who tries to make generali- zations about business success by studying ex- isting companies or managers falls into the classic statistical trap of selection bias—that is, of relying on samples that are not representa- tive of the whole population they’re studying. So if business researchers study only successful companies, any relationships they infer be- tween management practice and success will be necessarily misleading.

The theoretically correct way to discover what makes a business successful is to look at both thriving and floundering companies. Then business researchers will correctly iden- tify the qualities that separate the successes from the failures. Researchers might con- clude—as many have—that the strength of a company’s culture is associated with success because many successful companies have strong cultures. But if they were to study bank- rupt companies as well, they might find that many of those also had strong cultures. They might then be moved to hypothesize that the

For the exclusive use of S. Sadeghi, 2020.

This document is authorized for use only by Sohail Sadeghi in FE202 (A) Organizational Leadership taught by Koppman, S., University of California - Irvine from Sep 2020 to Mar 2021.

Selection Bias and the Perils of Benchmarking

•

HBR

ARGE

harvard business review • april 2005 page 2

Jerker Denrell

is an assistant professor of organizational behavior at Stanford Graduate School of Business in Stan- ford, California. A more-detailed discus- sion of the concepts in this article can be found in his paper, “Vicarious Learn- ing, Undersampling of Failure, and the Myths of Management,”

Organization Science,

May–June 2003.

nature

of a company’s culture is at least as im- portant as its

intensity

and then look more deeply into the whole issue of culture.

Similarly, if we want to examine effective leadership traits, we cannot look only at excel- lent managers. We must also consider manag- ers who failed to be promoted, were demoted, or were fired. Perhaps their styles of leadership were equally visionary—or humble. Without looking, we cannot tell.

The Blinding Light of Success

Selection bias is a difficult trap for business scholars and practitioners to avoid because good performance is rewarded by survival. Any sample of current managers will contain more successes than failures, if firms’ internal selection systems work properly. Similarly, poorly performing firms tend to fail and disap- pear, and so any sample of existing companies by definition consists largely of successful ones.

For that reason, managers are less likely to be infected with selection bias if they’re work- ing in an emerging industry. The evidence of failure is all around them. During the Internet boom, for instance, scores of new companies came into and went out of business. What’s more, many were able to stay afloat for some time with little, or even no, revenue. Managers trying to evaluate the merits of the various strategies at that stage of the online sector’s evolution could work off a relatively unbiased data set.

Of all managers, venture capitalists are per- haps the least likely to suffer from selection bias. Since only about 10% of all investments they make will become profitable, VCs invest in many different ventures in the hope that large returns on the few successes will compen- sate for the numerous losses. So VCs observe many failures, and their base of experience is almost completely unskewed by success.

It is when an industry matures and the fail- ure rate falls off that selection bias becomes a problem. After the dot-com crash, poorly per- forming companies finally went out of busi- ness, and fewer firms entered, which meant that not as many subsequently failed. At the same time, companies like Amazon, Google, and eBay grew larger, and even profitable, at- tracting more attention. Going forward, it is likely that only a few large firms will dominate in this industry, and the myriad companies

that have followed similar strategies but failed will be forgotten.

The effect of bias is almost certainly larger than most people think because the winnow- ing process in most industries is so dramatic. Some studies have shown that 50% of all new businesses fail during their first three to five years. Consider, for example, the U.S. tire in- dustry. After a period of rapid growth, the number of firms peaked in 1922 at 274. By 1936, there were just 49 survivors, a decline of more than 80%. So anyone studying the indus- try in the 1930s would have been able to ob- serve just a very small sample of the popula- tion that had originally entered.

That’s not to say that established companies don’t fail. They do, especially in the wake of radical shifts in technology or demand. But the fact is that people who work in established companies in mature industries are the most susceptible to selection bias. A regional mar- keting manager in a corporation like Coca-Cola or Procter & Gamble spends most of her time administering a successful brand and product line. She may have failed in implementing a new marketing practice at some time or an- other, but she will only ever have led the intro- duction of two or three new products and will probably never have started up a new business. In other words, her experience will be heavily biased toward success.

Selection bias isn’t just an issue for individ- ual companies. Judgments about general man- agement practices are also colored by it. A new quality program in one company may not work out as promised and be discontinued. Other firms in the same industry may succeed with the program and keep it. Unless you find out about the programs that failed, you will be able to observe only the successful cases.

I don’t mean to suggest that managers and analysts never study failures. But the ones they look at tend to be the really spectacular fias- coes or those, like Enron, that provoke strong moral outrage. And even then, it’s usually only in the moment. How many managers spend their time studying the corporate collapses of the 1980s? Yet they still read books about the manufacturing strategies of the Japanese inno- vators of that time.

Where the Dangers Lie

What kinds of traps do managers fall into when they rely on biased data? Three are

For the exclusive use of S. Sadeghi, 2020.

This document is authorized for use only by Sohail Sadeghi in FE202 (A) Organizational Leadership taught by Koppman, S., University of California - Irvine from Sep 2020 to Mar 2021.

Selection Bias and the Perils of Benchmarking

•

HBR

ARGE

harvard business review • april 2005 page 3

likely. Perhaps the most prevalent mistake is to

overvalue risky business practices. The prob- lem is easy to see in the exhibit “The Effects of Bias,” which illustrates what happens when trends are drawn from incomplete data fields.

The graphs plot the relationship between engaging in a risky organizational practice and subsequent corporate performance. The first graph records data from all companies that have ever implemented the risky practice, while the second one excludes companies that failed. As you might expect, the performance of firms that do not employ the practice at all is relatively stable. But the greater the degree to which firms engage in the practice, the wider the gap between the successful and unsuccess- ful companies becomes, as performance either spikes or plummets. On average, though, as the trend line shows, engaging in the risky practice somewhat

reduces

performance. Now suppose that we observed this indus-

try only after many of the worst-performing companies had gone out of business or had been acquired by other firms. In that case, we would have seen the successes but few of the failures associated with the risky practice. As a result, the observed association between the risky practice and performance would be

posi- tive,

as the second graph shows—the reverse of the true association.

Lee Fleming aptly illustrates this dynamic in his

Harvard Business Review

article “Perfecting Cross-Pollination” (September 2004). Fleming finds that, on average, the value of innovations coming out of diverse, cross-functional teams is lower than the value of innovations produced by teams of scientists whose backgrounds are similar to one another. But the innovations that the more heterogeneous teams produce tend to be either breakthroughs or dismal fail- ures. In fact, the distribution of the innovation values as the diversity of team members in- creases looks quite similar to what we see in our first graph.

In most instances, however, data on failed projects are not available, at least about failed projects in other companies, so most managers would be able to observe just the distribution pattern we see in our second graph. As a result, they would overestimate the value of cross- functional teams. Only by collecting data on both successes and failures, as Fleming did, could they spot the risks involved in using

The Effects of Bias

What happens when people draw conclusions from incomplete data? Suppose you are investigating the relationship between corporate performance and a particular risky business practice, such as using cross-functional teams. If you plotted the results of all companies that engaged in this practice (or any such practice), you would find that the more widespread the practice, the more vol- atile company performance, and your graph would look like this:

On average, as the trend line indicates, any such risky practice is correlated somewhat negatively with performance.

But suppose that you looked only at existing companies and excluded all those that had gone out of business while engaged in this practice. Then your graph would look like this:

Now the trend line will indicate a positive correlation, which of course is not the case. Thus selection bias will cause you to draw precisely the wrong conclusion.

zero performance

trend line

low

high

low high

Only Existing Companies

Prevalence of risky business practice

Company performance

Co py

rig ht

00 5

H ar

va rd

B us

in es

s Sc

ho ol

Pu bl

is hi

ng C

or po

ra tio

n. A

ll rig

ht s

re se

rv ed

zero performance

trend line

low highPrevalence of risky business practice

low

high

Company performance

All Companies

Co py

rig ht

00 5

H ar

va rd

B us

in es

s Sc

ho ol

Pu bl

is hi

ng C

or po

ra tio

n. A

ll rig

ht s

re se

rv ed

For the exclusive use of S. Sadeghi, 2020.

This document is authorized for use only by Sohail Sadeghi in FE202 (A) Organizational Leadership taught by Koppman, S., University of California - Irvine from Sep 2020 to Mar 2021.

Selection Bias and the Perils of Benchmarking

•

HBR

ARGE

harvard business review • april 2005 page 4

cross-functional research teams. (Another ex- ample of the same dynamic is described in the sidebar, “How Wrong Can You Get?”)

A second trap for unwary managers arises from the fact that performance often feeds on itself, so that current accomplishments are un- fairly magnified by past achievements. To see how this works, imagine that a company is a runner competing against other runners. If the runner wins ten independent races, he is prob- ably better than the others, who can learn from him. But suppose instead that the out- come of one race affects subsequent races. That is, if the runner wins by one minute in the first race, he gets a one-minute head start in the next race, and so on. Clearly, winning ten such races is less impressive, since a victory in the first race gives the runner a higher chance of winning the second, and an even higher chance of winning the third, and so on.

Many industries work the same way. For ex- ample, a telephone company or a software firm that had a large market share in 2004 will probably also have a large market share in 2005, owing to customer inertia and switching costs. Thus, even if managers do a poor job in 2005, such a company might still turn in high profits, as managers coast on their past accom- plishments or good luck.

Focusing on stock market returns instead of profits mitigates this problem, since changes in stock prices, in a well-functioning market, do reflect changes in performance. But defining success by stock market returns introduces

other problems. As Wharton professor Sidney Winter has pointed out, a company’s stock price will hold steady when one excellent CEO succeeds another. However, the share price will increase when the company exchanges an inferior CEO for a better, but still substandard, CEO. Maintaining excellence, in other words, might be less well rewarded than becoming merely mediocre.

A third problem with looking only at high performers for clues to high performance is the issue of reverse causality. Data may, for in- stance, reveal a strong association between the strength of a company’s culture and its perfor- mance. But does a strong culture lead to high performance or the other way around? The chicken-and-egg problem is especially knotty in this instance since high performance in itself affects corporate culture in several ways. To begin with, it’s probably easier to build a team- based culture in a healthy firm than in a failing one, where workers are likely to be demoral- ized and disloyal. High-performing companies also can afford to institute programs and prac- tices that low-performing firms cannot. Some of these expensive and time-consuming activi- ties might actually reduce performance at struggling companies.

What’s more, managers’ expectations of per- formance may influence their choice of strate- gies and thus confound interpretations of the effect those choices have. As William Boulding and Markus Christen observed in “First-Mover Disadvantage” (HBR, October 2001), compa- nies that have innovative products and strong distribution capabilities often choose to enter new markets early. Their strong products and capabilities produce high returns. As a result, however, their managers associate early entry with high performance in all cases, even when there is a first-mover disadvantage.

Bias, Bias Everywhere

Many of the popular theories on performance are riddled with selection bias. One of the most enduring ideas in management, for in- stance, is the notion that successful firms are those that focus most of their resources on one area or technology rather than diversifying. Books such as

In Search of Excellence, Built to Last,

and

Profit from the Core

all recommend that managers “stick to their knitting” and “fo- cus on the core.”

Typically, the research studies behind these

How Wrong Can You Get?

During World War II, the statistician Abraham Wald was assessing the vulner- ability of airplanes to enemy fire. All the available data showed that some parts of planes were hit disproportionately more often than other parts. Military person- nel concluded, naturally enough, that these parts should be reinforced. Wald, however, came to the opposite conclu- sion: The parts hit least often should be protected. His recommendation re- flected his insight into the selection bias inherent in the data, which represented only those planes that returned. Wald reasoned that a plane would be less

likely to return if it were hit in a critical area and, conversely, that a plane that did return even when hit had probably not been hit in a critical location. Thus, he argued, reinforcing those parts of the returned planes that sustained many hits would be unlikely to pay off.

1. The Wald story is one of the most widely cited anecdotes in the statistical community. To find out more about it, see W. Allen Wallis, “The Statis- tical Research Group, 1942–1945,”

Journal of the American Statistical Association,

June 1980, and M. Mangel and F.J. Samaniego, “Abraham Wald’s Work on Aircraft Survivability,”

Journal of the American Statistical Association,

June 1984.

For the exclusive use of S. Sadeghi, 2020.

This document is authorized for use only by Sohail Sadeghi in FE202 (A) Organizational Leadership taught by Koppman, S., University of California - Irvine from Sep 2020 to Mar 2021.

Selection Bias and the Perils of Benchmarking

•

HBR

ARGE

harvard business review • april 2005 page 5

books look only at existing companies or— even more narrowly—only at highly successful companies. As a result, their authors overesti- mate the benefits of focus. Consider, for exam- ple, Chris Zook and James Allen’s finding in

Profit from the Core

that 78% of all high-perfor- mance firms focused on one set of core activi- ties while only 22% of lower-performance firms did. The study comprised some 1,854 compa- nies, judging high-performance according to share price returns, sales, and profit ratios, but it included only businesses that survived throughout the study period. It did not, there- fore, consider any company that started with a focused strategy but then failed.

Including those failures would have changed the picture substantially. According to Zook and Allen, 13% of all firms achieved high perfor- mance, of which 78%—or 188 firms—focused on the core. If in that period just 200 other companies with focused strategies that had gone out of business had been included in the sample, then the true relationship between focus and performance would be the precise opposite of the one Zook and Allen infer.

Another fond notion often lauded by man- agement gurus and the popular press is that CEOs should be bold and take risks. Indeed, many stories in the business press celebrate the intuition of certain great leaders. No less an authority than Jack Welch entitled his autobi- ography

Straight from the Gut

. Some leaders— notably Sony’s Akio Morita—have gone so far as to eschew market research altogether, be- lieving their instincts are a better guide to mar- ket changes.

It’s certainly true that companies can be handsomely rewarded when their CEOs take big risks. Suppose you are operating in an in- dustry—fashion, say, or consumer electron- ics—where first movers have an advantage but where there is also considerable uncertainty re- garding consumer preferences. To gain first- mover advantage, a company must act quickly. The top-performing companies will be those that, led largely by the instincts of their senior managers, are lucky enough to launch prod- ucts that happen to appeal to customers.

But the worst-performing companies will also be those that act on hunches—and hap- pen to launch products that don’t appeal to customers. Since few people advertise their failures, and many of these unfortunate firms cease to exist, we hear mainly about the suc-

cess of decisions based on gut feelings and little about the countless “visionaries” who simi- larly tried to revolutionize industries but did not.

The point here is not that all the popular theories about performance are wrong. I don’t know. There may be a genuine link between success and focus. In some industries, the strength of a culture may matter regardless of its nature. And the instincts of some managers may be as sound a basis for strategic decision making as any amount of analysis. But what I do know is that no managers should accept a theory about business unless they can be confi- dent that the theory’s advocates are working off an unbiased data set.

Fixing the Problem

The most obvious step to take to guard against selection bias is to get all the data you can on failure. Within your organizations, you must in- sist that data on internal failures be systemati- cally collected and analyzed. Such information can otherwise easily disappear because the peo- ple responsible may leave the organization or be unwilling to talk. Looking outside your com- pany, you should extend your benchmarking exercises to include less-than-successful firms. Industry associations can help you collect data about failures of new practices and concepts.

Despite your best efforts, it’s unlikely you can ever be completely confident that your data are unbiased. Fortunately, you do have some backup, because economists and statisti- cians have developed a number of tools to cor- rect for selection bias. These tools, however, are grounded in certain assumptions, which may be more or less realistic, depending on the context.

Suppose, for example, that we want to esti- mate the average return on equity of all com- panies in a given industry, but we have avail- able only the ROE data of surviving firms. Since low-profit businesses are more likely to fail, just taking the average ROE of all surviv- ing firms would lead to too high an estimate. But suppose we assume that ROE is distributed along a standard bell curve and that all busi- nesses with a negative ROE will fail. Then we can use the data we have to estimate the aver- age ROE for all firms because the information on hand is enough to tell us how steep the curve is, how broad, and what the average is.

This approach can be used to correct for

How many managers study the corporate collapses of the 1980s? Yet they still read about the Japanese manufacturing strategies of that time.

For the exclusive use of S. Sadeghi, 2020.

This document is authorized for use only by Sohail Sadeghi in FE202 (A) Organizational Leadership taught by Koppman, S., University of California - Irvine from Sep 2020 to Mar 2021.

Selection Bias and the Perils of Benchmarking

•

HBR

ARGE

harvard business review • april 2005 page 6

bias in any situation in which we can apply formal statistical tools. For instance, let’s say we suspect that in a particular industry, the more training a company’s sales staff gets, the higher the average salesperson’s performance will be and the more consistent the entire sales staff’s performance will be. Suppose fur- ther that we have detailed data on the invest- ments in training made by most firms cur- rently operating in the industry. If we can safely assume that performance follows some specified distribution pattern, we can in prin- ciple use the data we have to obtain an unbi- ased estimate of how investments in training actually do influence the average level and variability of sales staff performance, even if we do not have data on firms that failed. Es- sentially, what we are doing is inferring the shape of a particular iceberg by observing its tip and making (we hope) a reasonable as- sumption about the relationship between the tips of icebergs and the rest of them.

The pioneer of these statistical methods was James Tobin, winner of the 1981 Nobel Prize in economics. His work was later built upon by James Heckman, who himself received a Nobel Prize in 2000 for his contributions in this area. In recent years, management schol- ars have applied these methods to correct for

selection bias in their own research and have started to advocate for their use in the broader managerial community. In “Getting the Most out of All Your Customers” (HBR, July–August 2004), for instance, Jacquelyn S. Thomas, Werner Reinartz, and V. Kumar demonstrate how such tools can be used to improve the cost-effectiveness of marketing investments.

Cautionary words and counsels of failure, I know, are seldom well received. Managers crave certainties and role models from busi- ness literature, and to some extent they have to. They live in a fast-paced world, and they often cannot afford to postpone action until they get better data. But there really is no ex- cuse for ignoring the glaring traps we’ve de- scribed in these pages. Success may be more in- spirational, but the inescapable logic of statistics dictates that managers in pursuit of high performance are more likely to attain their goal if they give the stories of their com- petitors’ failures as full a hearing as they cur- rently do the stories of their successes.

Reprint R0504H

To order, see the next page or call 800-988-0886 or 617-783-7500 or go to www.hbr.org

For the exclusive use of S. Sadeghi, 2020.

This document is authorized for use only by Sohail Sadeghi in FE202 (A) Organizational Leadership taught by Koppman, S., University of California - Irvine from Sep 2020 to Mar 2021.

http://harvardbusinessonline.hbsp.harvard.edu/relay.jhtml?name=itemdetail&referral=4320&id=R0504H

http://www.hbr.org