INS630
analele
why_quants_dont_know_everything.pdf
3/24/2014 Why Quants Don’t Know Everything | Wired Business | Wired.com
http://www.wired.com/business/2014/01/quants-dont-know-everything/ 1/14
The reason the number-loving quants win is they’re almost always right. But what happens after they win is not always the data-driven paradise they and their boosters expect.
Business Peo ple and Tec hno lo gies That Matter
Big Data
Follow @felixsalm on
Why Quants Don’t Know Everything BY FELIX SALMON
01.07.14
9:30 AM Share on Facebook 1.1k shares
Tw eet 754 80 Share 195 5
3/24/2014 Why Quants Don’t Know Everything | Wired Business | Wired.com
http://www.wired.com/business/2014/01/quants-dont-know-everything/ 3/14
Lamosca
By now, nearly everyone from the president of the United States on down has admitted
that the National Security Agency went too far. Documents leaked by Edward Snowden,
the rogue NSA contractor who has since gained asylum in Russia, paint a picture of an
organization with access to seemingly every word typed or spoken on any electronic
device, anywhere in the world. And when news of the NSA’s reach became public—as it
was surely bound to do at some point—the entire US intelligence apparatus was thrust
into what The New York Times recently called a “crisis of purpose and legitimacy.”
It was a crisis many years in the making. Over the course of three decades, the NSA
slowly transformed itself from the nation’s junior spy agency to the centerpiece of the
entire intelligence system. As the amount of data in the world doubled, and doubled
again, and again, the NSA kept up with it—even as America’s human intelligence
capability, as typified by old-fashioned CIA spies in the field, struggled to do anything
useful with the unprecedented quantities of signals intelligence they had access to.
Trained agency linguists capable of parsing massive quantities of Arabic- and Farsi-
language intercepts don’t scale up nearly as easily as data centers do.
That, however, wasn’t the computer geeks’ problem. Once it was clear that the NSA
could do something, it seemed inarguable that the agency should do it—even after the
bounds of information overload (billions of records added to bulging databases every
day) or basic decency (spying on allied heads of state, for example) had long since been
surpassed. The value of every marginal gigabyte of high tech signals intelligence was,
at least in theory, quantifiable. The downside—the inability to prioritize essential
intelligence and act on it; the damage to America’s democratic legitimacy—was not. As
a result, during the past couple of decades spycraft went from being a pursuit driven by
human judgment calls to one driven by technical capability.
3/24/2014 Why Quants Don’t Know Everything | Wired Business | Wired.com
http://www.wired.com/business/2014/01/quants-dont-know-everything/ 4/14
COMMENTATORS WHO “TRUSTED THEIR GUT” ABOUT MITT ROMNEY HAD THEIR GUT KICKED BY NATE SILVER, THE STATS WHIZ WHO CALLED THE ELECTION FOR OBAMA.
This shift in US intelligence mirrors a definite pattern of the past 30 years, one that we
can see across fields and institutions. It’s the rise of the quants—that is, the ascent to
power of people whose native tongue is numbers and algorithms and systems rather
than personal relationships or human intuition. Michael Lewis’ Moneyball vividly
recounts how the quants took over baseball, as statistical analysis trumped traditional
scouting and propelled the underfunded Oakland A’s to a division-winning 2002
season. More recently we’ve seen the rise of the quants in politics. Commentators who
“trusted their gut” about Mitt Romney’s chances had their gut kicked by Nate Silver,
the stats whiz who called the election days beforehand as a lock for Obama, down to
the very last electoral vote in the very last state.
The reason the quants win is that they’re almost always right—at least at first. They
find numerical patterns or invent ingenious algorithms that increase profits or solve
problems in ways that no amount of subjective experience can match. But what
happens after the quants win is not always the data-driven paradise that they and their
boosters expected. The more a field is run by a system, the more that system creates
incentives for everyone (employees, customers, competitors) to change their behavior
in perverse ways—providing more of whatever the system is designed to measure and
produce, whether that actually creates any value or not. It’s a problem that can’t be
solved until the quants learn a little bit from the old-fashioned ways of thinking
they’ve displaced.
No matter the discipline or industry, the rise of the quants tends to happen in four
stages. Stage one is what you might call pre-disruption, and it’s generally best visible
in hindsight. Think about quaint dating agencies in the days before the arrival of Match
.com and all the other algorithm-powered online replacements. Or think about retail in
the era before floor-space management analytics helped quantify exactly which goods
ought to go where. For a live example, consider Hollywood, which, for all the money it
spends on market research, is still run by a small group of lavishly compensated studio
3/24/2014 Why Quants Don’t Know Everything | Wired Business | Wired.com
http://www.wired.com/business/2014/01/quants-dont-know-everything/ 5/14
executives, all of whom are well aware that the first rule of Hollywood, as memorably
summed up by screenwriter William Goldman, is “Nobody knows anything.” On its face,
Hollywood is ripe for quantification—there’s a huge amount of data to be mined,
considering that every movie and TV show can be classified along hundreds of different
axes, from stars to genre to running time, and they can all be correlated to box office
receipts and other measures of profitability.
Next comes stage two, disruption. In most industries, the rise of the quants is a recent
phenomenon, but in the world of finance it began back in the 1980s. The unmistakable
sign of this change was hard to miss: the point at which you started getting targeted
and personalized offers for credit cards and other financial services based not on the
relationship you had with your local bank manager but on what the bank’s algorithms
deduced about your finances and creditworthiness. Pretty soon, when you went into a
branch to inquire about a loan, all they could do was punch numbers into a computer
and then give you the computer’s answer.
For a present-day example of disruption, think about politics. In the 2012 election,
Obama’s old-fashioned campaign operatives didn’t disappear. But they gave money
and freedom to a core group of technologists in Chicago—including Harper Reed,
former CTO of the Chicago-based online retailer Threadless—and allowed them to
make huge decisions about fund-raising and voter targeting. Whereas earlier
campaigns had tried to target segments of the population defined by geography or
demographic profile, Obama’s team made the campaign granular right down to the
individual level. So if a mom in Cedar Rapids was on the fence about who to vote for, or
whether to vote at all, then instead of buying yet another TV ad, the Obama campaign
would message one of her Facebook friends and try the much more effective personal
approach.
Oakland, 2002—when data geeks moneyballed the national pastime. Dave Kaup/Getty Images
Most strikingly, the campaign perfected the art of A/B testing—the practice of testing
alternate versions—when it came to fund-raising emails. Writing effective language for
such appeals, and designing them in the best possible manner, has traditionally been
considered an art—but all gut intuition was discarded in favor of raw data about what
3/24/2014 Why Quants Don’t Know Everything | Wired Business | Wired.com
http://www.wired.com/business/2014/01/quants-dont-know-everything/ 6/14
worked and what didn’t. One email in June, headlined “I will be outspent,” raised $2.6
million on its own, while projections based on their testing indicated that other emails
would have raised less than a fifth of that. Everything was A/B tested, from the
background color (yellow worked better than white for some reason) to the greeting,
the subject line, and the size of the request.
After disruption, though, there comes at least some version of stage three: overshoot.
The most common problem is that all these new systems—metrics, algorithms,
automated decisionmaking processes—result in humans gaming the system in rational
but often unpredictable ways. Sociologist Donald T. Campbell noted this dynamic back
in the ’70s, when he articulated what’s come to be known as Campbell’s law: “The more
any quantitative social indicator is used for social decision-making,” he wrote, “the
more subject it will be to corruption pressures and the more apt it will be to distort and
corrupt the social processes it is intended to monitor.”
On a managerial level, once the quants come into an industry and disrupt it, they often
don’t know when to stop. They tend not to have decades of institutional knowledge
about the field in which they have found themselves. And once they’re empowered,
quants tend to create systems that favor something pretty close to cheating. As soon
as managers pick a numerical metric as a way to measure whether they’re achieving
their desired outcome, everybody starts maximizing that metric rather than doing the
rest of their job—just as Campbell’s law predicts.
ONCE QUANTS DISRUPT AN INDUSTRY, THEY OFTEN DON’T KNOW WHEN TO STOP—AND THEY CREATE SYSTEMS THAT ENCOURAGE CHEATING.
Policing is a good example, as explained by Harvard sociologist Peter Moskos in his
book Cop in the Hood: My Year Policing Baltimore’s Eastern District. Most cops have a
pretty good idea of what they should be doing, if their goal is public safety: reducing
crime, locking up kingpins, confiscating drugs. It involves foot patrols, deep
investigations, and building good relations with the community. But under statistically
driven regimes, individual officers have almost no incentive to actually do that stuff.
Instead, they’re all too often judged on results—specifically, arrests. (Not even
3/24/2014 Why Quants Don’t Know Everything | Wired Business | Wired.com
http://www.wired.com/business/2014/01/quants-dont-know-everything/ 7/14
convictions, just arrests: If a suspect throws away his drugs while fleeing police, the
police will chase and arrest him just to get the arrest, even when they know there’s no
chance of a conviction.)
The same goes for the rise of “teaching to the test” in public schools, or the perverse
incentives placed on snowplow operators, who, paid by the quantity of snow cleared,
might simply ignore patches of lethal black ice. Even with the 2012 Obama campaign, it
became hard to learn about the candidate’s positions by visiting his website, because it
was so optimized for maximizing donations—an easy and obvious numerical target—
that all other functions fell by the wayside.
The most profound example of overshoot, of course, happened in finance, where the
rise of quantification could concentrate decisionmaking—and moneymaking—within a
relatively small group of people at a bank’s headquarters. Soon they were trying to
optimize their algorithms to maximize profit, minimize risk, and make millions of
dollars for themselves. Global regulators didn’t help: In 2004, in sympathy with the
over-leveraged, hyper-quantified banking system, the Basel Committee—the
Switzerland-based body that oversees world finance—put out the Basel II accord, more
than 250 pages of regulations that effectively placed individual banks in the driver’s
seat. The accord essentially embraced all of the quantitative techniques used by the
wizards who would end up blowing up Wall Street, and it allowed banks to operate with
astonishingly high levels of debt. As everybody knows, all of that ended in catastrophe
in 2008. (You can read more about the particular math of that cataclysm in my March
2009 cover story for WIRED, “A Formula for Disaster.”)
It’s increasingly clear that for smart organizations, living by numbers alone simply won’t
work. That’s why they arrive at stage four: synthesis—the practice of marrying
quantitative insights with old-fashioned subjective experience. Nate Silver himself has
written thoughtfully about examples of this in his book, The Signal and the Noise. He
cites baseball, which in the post-Moneyball era adopted a “fusion approach” that leans
on both statistics and scouting. Silver credits it with delivering the Boston Red Sox’s
first World Series title in 86 years. Or consider weather forecasting: The National
Weather Service employs meteorologists who, understanding the dynamics of weather
systems, can improve forecasts by as much as 25 percent compared with computers
3/24/2014 Why Quants Don’t Know Everything | Wired Business | Wired.com
http://www.wired.com/business/2014/01/quants-dont-know-everything/ 8/14
Felix Salmon is the finance blogger at Reuters. Any views expressed may or may not be his own, but in any case are very unlikely to be those of his employer. Read more by Felix Salmon Follow @felixsalmon and @felixreuters on Twitter.
alone. A similar synthesis holds in economic forecasting: Adding human judgment to
statistical methods makes results roughly 15 percent more accurate. And it’s even true
in chess: While the best computers can now easily beat the best humans, they can in
turn be beaten by humans aided by computers.
In finance too we’re starting to see at least the outlines of a synthesis. In September
2010, the Basel Committee came out with Basel III, and while it doesn’t fully dismantle
Basel II, it does add layers of common sense on top of all the rocket science. As well as
raising the required capital ratio, it sets a leverage ratio (effectively a maximum size
that a bank can grow to, given the amount of capital it has) and liquidity requirements
that experienced bankers know create a cushion for the whole system. Essentially,
while the algorithms were given free rein under Basel II, there’s a host of overrides in
Basel III that put power back where it belongs, in the hands of experienced regulators.
Basel III isn’t perfect, but no international system of bank regulation could ever hope
to be. In a few years’ time, if and when it gets fully implemented, it’s going to be a vast
improvement on what preceded it.
That’s what a good synthesis of big data and human intuition tends to look like. As
long as the humans are in control, and understand what it is they’re controlling, we’re
fine. It’s when they become slaves to the numbers that trouble breaks out. So let’s
celebrate the value of disruption by data—but let’s not forget that data isn’t
everything.
