1500

Chapter Title: INTRODUCTION THE NEED TO KNOW Book Title: The Black Box Society

Book Subtitle: The Secret Algorithms That Control Money and Information

Book Author(s): FRANK PASQUALE

Published by: Harvard University Press

Stable URL: https://www.jstor.org/stable/j.ctt13x0hch.3

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected]. Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at https://about.jstor.org/terms

Harvard University Press is collaborating with JSTOR to digitize, preserve and extend access to The Black Box Society

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

1 INTRODUCTION—THE NEED

TO KNOW

EV ERY BODY K NOWS the story about the man crawling intently around a lamppost on a dark night. When a police offi cer comes along and wants to know what he’s doing, he says he’s looking for his keys. “You lost them here?” asks the cop. “No,” the seeker replies, “but this is where the light is.” This bromide about futility has lately taken on a whole new meaning as a meta phor for our increasingly enigmatic technologies.

There’s a noble tradition among social scientists of trying to clar- ify how power works: who gets what, when, where, and why.1 Our common life is explored in books like The Achieving Society, The

Winner-Take-All Society, The Good Society, and The Decent Society. At their best, these works also tell us why such inquiry matters.2

But efforts like these are only as good as the information available. We cannot understand, or even investigate, a subject about which nothing is known. Amateur epistemologists have many names for this problem. “Unknown unknowns,” “black swans,” and “deep se- crets” are pop u lar catchphrases for our many areas of social blank- ness.3 There is even an emerging fi eld of “agnotology” that studies the “structural production of ignorance, its diverse causes and con- formations, whether brought about by neglect, forgetfulness, myopia, extinction, secrecy, or suppression.” 4

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

2 T H E B L A C K B O X S O C I E T Y

Gaps in knowledge, putative and real, have powerful implica- tions, as do the uses that are made of them. Alan Greenspan, once the most powerful central banker in the world, claimed that today’s markets are driven by an “unredeemably opaque” version of Adam Smith’s “invisible hand,” and that no one (including regulators) can ever get “more than a glimpse at the internal workings of the sim- plest of modern fi nancial systems.” If this is true, libertarian policy would seem to be the only reasonable response. Friedrich von Hayek, a preeminent theorist of laissez- faire, called the “knowledge prob- lem” an insuperable barrier to benevolent government interventions in the economy.5

But what if the “knowledge problem” is not an intrinsic aspect of the market, but rather is deliberately encouraged by certain busi- nesses? What if fi nanciers keep their doings opaque on purpose, pre- cisely to avoid or to confound regulation? That would imply some- thing very different about the merits of deregulation.

The challenge of the “knowledge problem” is just one example of a general truth: What we do and don’t know about the social (as op- posed to the natural) world is not inherent in its nature, but is itself a function of social constructs. Much of what we can fi nd out about companies, governments, or even one another, is governed by law. Laws of privacy, trade secrecy, the so- called Freedom of Informa- tion Act— all set limits to inquiry. They rule certain investigations out of the question before they can even begin. We need to ask: To whose benefi t?

Some of these laws are crucial to a decent society. No one wants to live in a world where the boss can tape our bathroom breaks. But the laws of information protect much more than personal privacy. They allow pharmaceutical fi rms to hide the dangers of a new drug behind veils of trade secrecy and banks to obscure tax liabilities be- hind shell corporations. And they are much too valuable to their benefi ciaries to be relinquished readily.

Even our po liti cal and legal systems, the spaces of our common life that are supposed to be the most open and transparent, are be- ing colonized by the logic of secrecy. The executive branch has been lobbying ever more forcefully for the right to enact and enforce “se- cret law” in its pursuit of the “war on terror,” and voters contend in

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

I N T R O D U C T I O N — T H E N E E D T O K N O W 3

an electoral arena fl ooded with “dark money”— dollars whose do- nors, and whose infl uence, will be disclosed only after the election, if at all.6

But while powerful businesses, fi nancial institutions, and govern- ment agencies hide their actions behind nondisclosure agreements, “proprietary methods,” and gag rules, our own lives are increasingly open books. Everything we do online is recorded; the only ques- tions left are to whom the data will be available, and for how long. Anonymizing software may shield us for a little while, but who knows whether trying to hide isn’t itself the ultimate red fl ag for watchful authorities? Surveillance cameras, data brokers, sensor net- works, and “supercookies” record how fast we drive, what pills we take, what books we read, what websites we visit. The law, so aggressively protective of secrecy in the world of commerce, is in- creasingly silent when it comes to the privacy of persons.

That incongruity is the focus of this book. How has secrecy be- come so important to industries ranging from Wall Street to Silicon Valley? What are the social implications of the invisible practices that hide the way people and businesses are labeled and treated? How can the law be used to enact the best possible balance between privacy and openness? To answer these questions is to chart a path toward a more intelligible social order.

But fi rst, we must fully understand the problem. The term “black box” is a useful meta phor for doing so, given its own dual meaning. It can refer to a recording device, like the data- monitoring systems in planes, trains, and cars. Or it can mean a system whose workings are mysterious; we can observe its inputs and outputs, but we cannot tell how one becomes the other. We face these two meanings daily: tracked ever more closely by fi rms and government, we have no clear idea of just how far much of this information can travel, how it is used, or its consequences.7

The Power of Secrecy

Knowledge is power. To scrutinize others while avoiding scrutiny oneself is one of the most important forms of power.8 Firms seek out intimate details of potential customers’ and employees’ lives, but give regulators as little information as they possibly can about

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

4 T H E B L A C K B O X S O C I E T Y

their own statistics and procedures.9 Internet companies collect more and more data on their users but fi ght regulations that would let those same users exercise some control over the resulting digi- tal dossiers.

As technology advances, market pressures raise the stakes of the data game. Surveillance cameras become cheaper every year; sensors are embedded in more places.10 Cell phones track our movements; programs log our keystrokes. New hardware and new software prom- ise to make “quantifi ed selves” of all of us, whether we like it or not.11 The resulting information— a vast amount of data that until recently went unrecorded— is fed into databases and assembled into profi les of unpre ce dented depth and specifi city.

But to what ends, and to whose? The decline in personal privacy might be worthwhile if it were matched by comparable levels of trans- parency from corporations and government. But for the most part it is not. Credit raters, search engines, major banks, and the TSA take in data about us and convert it into scores, rankings, risk calculations, and watch lists with vitally important consequences. But the propri- etary algorithms by which they do so are immune from scrutiny, except on the rare occasions when a whistleblower litigates or leaks.

Sometimes secrecy is warranted. We don’t want terrorists to be able to evade detection because they know exactly what Homeland Security agents are looking out for.12 But when every move we make is subject to inspection by entities whose procedures and personnel are exempt from even remotely similar treatment, the promise of democracy and free markets rings hollow. Secrecy is approaching critical mass, and we are in the dark about crucial decisions. Greater openness is imperative.

Reputation, Search, Finance

At the core of the information economy are Internet and fi nance companies that accumulate vast amounts of digital data, and with it intimate details of their customers’— our—lives. They use it to make important decisions about us and to infl uence the decisions we make for ourselves. But what do we know about them? A bad credit score may cost a borrower hundreds of thousands of dollars, but he will never understand exactly how it was calculated. A predictive

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

I N T R O D U C T I O N — T H E N E E D T O K N O W 5

analytics fi rm may score someone as a “high cost” or “unreliable” worker, yet never tell her about the decision.

More benignly, perhaps, these companies infl uence the choices we make ourselves. Recommendation engines at Amazon and You- Tube affect an automated familiarity, gently suggesting offerings they think we’ll like. But don’t discount the signifi cance of that “per- haps.” The economic, po liti cal, and cultural agendas behind their suggestions are hard to unravel. As middlemen, they specialize in shifting alliances, sometimes advancing the interests of customers, sometimes suppliers: all to orchestrate an online world that maxi- mizes their own profi ts.

Financial institutions exert direct power over us, deciding the terms of credit and debt. Yet they too shroud key deals in impenetrable layers of complexity. In 2008, when secret goings- on in the money world provoked a crisis of trust that brought the banking system to the brink of collapse, the Federal Reserve intervened to stabilize things— and kept key terms of those interventions secret as well. Journalists didn’t uncover the massive scope of its interventions until late 2011.13 That was well after landmark fi nancial reform legisla- tion had been debated and passed—without informed input from the electorate— and then watered down by the same corporate titans whom the Fed had just had to bail out.

Reputation. Search. Finance. These are the areas in which Big Data looms largest in our lives. But too often it looms invisibly, under- mining the openness of our society and the fairness of our markets. Consider just a few of the issues raised by the new technologies of ranking and evaluation:

• Should a credit card company be entitled to raise a couple’s interest rate if they seek marriage counseling? If so, should cardholders know this?

• Should Google, Apple, Twitter, or Facebook be able to shut out websites or books entirely, even when their content is com- pletely legal? And if they do, should they tell us?

• Should the Federal Reserve be allowed to print unknown sums of money to save banks from their own scandalous behavior? If so, how and when should citizens get to learn what’s going on?

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

6 T H E B L A C K B O X S O C I E T Y

• Should the hundreds of thousands of American citizens placed on secret “watch lists” be so informed, and should they be given the chance to clear their names?

The leading fi rms of Wall Street and Silicon Valley are not alone in the secretiveness of their operations, but I will be focusing pri- marily on them because of their unique roles in society. While ac- counting for “less than 10% of the value added” in the U.S. economy in the fourth quarter of 2010, the fi nance sector took 29 percent— $57.7 billion— of profi ts.14 Silicon Valley fi rms are also remarkably profi table, and powerful.15 What fi nance fi rms do with money, lead- ing Internet companies do with attention. They direct it toward some ideas, goods, and ser vices, and away from others. They or ga nize the world for us, and we have been quick to welcome this data- driven con ve nience. But we need to be honest about its costs.

Secrecy and Complexity

Deconstructing the black boxes of Big Data isn’t easy. Even if they were willing to expose their methods to the public, the modern Internet and banking sectors pose tough challenges to our under- standing of those methods. The conclusions they come to— about the productivity of employees, or the relevance of websites, or the attractiveness of investments— are determined by complex for- mulas devised by legions of engineers and guarded by a phalanx of lawyers.

In this book, we will be exploring three critical strategies for keeping black boxes closed: “real” secrecy, legal secrecy, and obfus- cation. Real secrecy establishes a barrier between hidden content and unauthorized access to it. We use real secrecy daily when we lock our doors or protect our e-mail with passwords. Legal secrecy obliges those privy to certain information to keep it secret; a bank employee is obliged both by statutory authority and by terms of employment not to reveal customers’ balances to his buddies.16 Obfuscation in- volves deliberate attempts at concealment when secrecy has been compromised. For example, a fi rm might respond to a request for information by delivering 30 million pages of documents, forcing its investigator to waste time looking for a needle in a haystack.17 And

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

I N T R O D U C T I O N — T H E N E E D T O K N O W 7

the end result of both types of secrecy, and obfuscation, is opacity, my blanket term for remediable incomprehensibility.18

Detailed investment prospectuses, for instance, can run to doz- ens or hundreds of pages. They can refer to other documents, and those to still others. There may be confl icts among the documents that the original source references.19 Anyone really trying to under- stand the investment is likely to have to pro cess thousands of pages of complicated legal verbiage— some of which can be quite obfusca- tory. The same holds for accounting statements. When law profes- sor Frank Partnoy and Pulitzer Prize– winning journalist Jesse Eis- inger teamed up to explore “what’s inside America’s banks” in early 2013, they were aghast at the enduring opacity. They reported on the banks as “ ‘black boxes’ that may still be concealing enormous risks— the sort that could again take down the economy.”20 Several quotes in the article portrayed an American banking system still out of control fi ve years after the crisis:

• “There is no major fi nancial institution today whose fi nancial statements provide a meaningful clue” about its risks, said one hedge fund manager.

• “After serving on the [Financial Accounting Standards] board [FASB],” said Don Young, “I no longer trust bank accounting.”

• Another former FASB member, asked if he trusted bank accounting, answered: “Absolutely not.”21

These quotes came fi ve years after the fi nancial crisis and three years after the Dodd- Frank Act, a gargantuan piece of legislation that comprehensively altered banking law. Financial crises result when a critical mass of investors act on that distrust, and their skep- ticism cascades throughout the system. And when governments step in with their “bailouts” and “liquidity facilities,” they add new layers of complexity to an already byzantine situation.

In the case of technology companies, complexity is not as impor- tant as secrecy. However sprawling the web becomes, Google’s search engineers are at least working on a “closed system”; their own company’s copies of the Internet. Similarly, those in charge of Twitter and Facebook “feeds” have a set body of information to

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

8 T H E B L A C K B O X S O C I E T Y

work with. Their methods are hard to understand primarily because of a mix of real and legal secrecy, and their scale. Interlocking technical and legal prohibitions prevent anyone outside such a com- pany from understanding fundamental facts about it.

Activists often press for transparency as a solution to the black box issues raised in this book. In many cases, sunshine truly is the “best disinfectant.” However, transparency may simply provoke complexity that is as effective at defeating understanding as real or legal secrecy. Government has frequently stepped in to require dis- closure and “plain language” formats for consumers. But fi nanciers have parried transparency rules with more complex transactions. When this happens, without substantial gains in effi ciency, regula- tors should step in and limit complexity. Transparency is not just an end in itself, but an interim step on the road to intelligibility.

The Secret Judgments of Software

So why does this all matter? It matters because authority is increas- ingly expressed algorithmically.22 Decisions that used to be based on human refl ection are now made automatically. Software encodes thousands of rules and instructions computed in a fraction of a sec- ond. Such automated pro cesses have long guided our planes, run the physical backbone of the Internet, and interpreted our GPSes. In short, they improve the quality of our daily lives in ways both noticeable and not.

But where do we call a halt? Similar protocols also infl uence— invisibly—not only the route we take to a new restaurant, but which restaurant Google, Yelp, OpenTable, or Siri recommends to us. They might help us fi nd reviews of the car we drive. Yet choosing a car, or even a restaurant, is not as straightforward as optimizing an engine or routing a drive. Does the recommendation engine take into account, say, whether the restaurant or car company gives its workers health benefi ts or maternity leave? Could we prompt it to do so? In their race for the most profi table methods of mapping so- cial reality, the data scientists of Silicon Valley and Wall Street tend to treat recommendations as purely technical problems. The values and prerogatives that the encoded rules enact are hidden within black boxes.23

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

I N T R O D U C T I O N — T H E N E E D T O K N O W 9

The most obvious question is: Are these algorithmic applications fair? Why, for instance, does YouTube (owned by Google) so con- sistently beat out other video sites in Google’s video search re- sults? How does one par tic u lar restaurant or auto stock make it to the top of the hit list while another does not? What does it mean when Internet retailers quote different prices for the same product to different buyers? Why are some borrowers cut slack for a late payment, while others are not?

Defenders of the status quo say that results like these refl ect a company’s good- faith judgment about the quality of a website, an in- vestment, or a customer. Detractors contend that they cloak self- serving appraisals and confl icts of interest in a veil of technologi- cal wizardry. Who is right? It’s anyone’s guess, as long as the algorithms involved are kept secret. Without knowing what Google actually does when it ranks sites, we cannot assess when it is acting in good faith to help users, and when it is biasing results to favor its own commercial interests. The same goes for status updates on Facebook, trending topics on Twitter, and even network management practices at telephone and cable companies. All these are protected by laws of secrecy and technologies of obfuscation.

The One- Way Mirror

With so much secrecy so publicly in place, it is easy for casual ob- servers to conclude that there is a rough parity between the infor- mational protection of individuals and civil associations and those of corporations and government. It is comforting to think that our personal bank rec ords are as secure as the bank’s own secrets. But I will attempt to overthrow this assumption. We do not live in a peaceable kingdom of private walled gardens; the contemporary world more closely resembles a one- way mirror. Important corpo- rate actors have unpre ce dented knowledge of the minutiae of our daily lives, while we know little to nothing about how they use this knowledge to infl uence the important decisions that we— and they— make.

Furthermore, even as critical power over money and new media rapidly concentrates in a handful of private companies, we remain largely ignorant of critical ways in which these companies interact

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

10 T H E B L A C K B O X S O C I E T Y

(and confl ict) with public powers. Though this book is primarily about the private sector, I have called it The Black Box Society (rather than The Black Box Economy) because the distinction between state and market is fading. We are increasingly ruled by what former po- liti cal insider Jeff Connaughton called “The Blob,” a shadowy net- work of actors who mobilize money and media for private gain, whether acting offi cially on behalf of business or of government.24 In one policy area (or industry) after another, these insiders decide the distribution of society’s benefi ts (like low- interest credit or secure employment) and burdens (like audits, wiretaps, and precarity).

Admittedly, as Jon Elster has written in his book Local Justice, there is no perfectly fair way to allocate opportunities.25 But a market- state increasingly dedicated to the advantages of speed and stealth crowds out even the most basic efforts to make these choices fairer. Tech- nocrats and managers cloak contestable value judgments in the garb of “science”: thus the insatiable demand for mathematical models that reframe subtle and subjective conclusions (such as the worth of a worker, ser vice, article, or product) as the inevitable dictate of salient, mea sur able data.26 Big data driven decisions may lead to unpre ce- dented profi ts. But once we use computation not merely to exercise power over things, but also over people, we need to develop a much more robust ethical framework than “the Blob” is now willing to entertain.

The Secrecy of Business and the Business of Secrecy

Today’s fi nance and Internet companies feverishly sort, rank, and rate. They say they keep techniques strictly secret in order to preserve valuable intellectual property— but their darker motives are also ob- vious. For example, litigation has revealed that some drug companies have cherry- picked the most positive studies for publication, hiding those with serious health or safety implications.27 Journalists are pry- ing open Wall Street’s pre- fi nancial crisis black boxes to this day.28 The Sunlight Foundation, Center for Effective Government, AllTri- als.net, and Transparency International press for openness.

Politicians are responding, and try to improve disclosure here and there. But they must be cautious. When a gadfl y proves too incon ve-

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

I N T R O D U C T I O N — T H E N E E D T O K N O W 11

nient, companies can band together in a super PAC, funding attacks on the would- be reformer without having to reveal what they are doing until well after the election.29

Asked about Google’s privacy practices, former CEO Eric Schmidt once said that “Google policy is to get right up to the creepy line and not cross it.” It is probably more accurate to say that he and other Sili- con Valley leaders don’t want to be caught crossing the creepy line.30 As long as secrecy can be used to undermine market competition and law enforcement, they will be emboldened to experiment with ever creepier, more intrusive, and even exploitative practices.

Looking Back

The quest for a more transparent society— more easily understood, and more open about its priorities— has animated leading reformers in the United States. Louis Brandeis’s comment that “sunlight is said to be the best of disinfectants,” so often cited today, is a century old, dating back to business scandals of the Gilded Age eerily simi- lar to today’s casino capitalism.31 Muckraking journalists and trust- busters of the Progressive Era shamed robber barons by exposing their misdeeds.32 They targeted politicians, too: the Publicity Act of 1910 mandated disclosure of campaign donations.33

Many states of the time took up similar reforms. Voters wanted politics and business subject to public scrutiny. After shady com- mercial practices surged again in the 1920s, the New Deal echoed and amplifi ed Progressivism. Congress, disgusted by the hucksters who paved the way for the great crash of 1929, imposed sweeping new disclosure obligations in the Securities Act of 1933 and the Se- curities Exchange Act of 1934. New legislation created the Federal Communications Commission and gave it plenary power to investi- gate abuses in the telegraph and radio industries.34 New Deal agen- cies revealed the inner workings of critical industries.35

Government balanced these new powers by opening itself up in important ways. For example, the Administrative Procedure Act (APA) of 1947 forced agencies to give the public notice and a chance to comment before they imposed important rules. Reformers built on the APA with the 1966 Freedom of Information Act, which opened up many government rec ords.36

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

12 T H E B L A C K B O X S O C I E T Y

In the 1960s, a broad co ali tion of interests fought both govern- ment and corporate secrecy in the name of citizen empowerment and consumer protection.37 Perhaps their most enduring legacy was the establishment of procedures of openness. For example, the Na- tional Environmental Policy Act required major federal projects to include Environmental Impact Statements that would reveal likely effects on air, water, fl ora, and fauna. Agencies ranging from the Food and Drug Administration to the Consumer Product Safety Commission now make daily activities less dangerous by revealing the risks of things we purchase.38

But there was always pushback. By the late 1960s, businesses were successfully challenging scrutiny from what they branded the “nanny state.” When the Environmental Protection Agency wanted to release data on the composition of some pesticides, for example, Monsanto fought back. It won a Supreme Court ruling that pre- vented the disclosure on the grounds that the formulations were a “trade secret” (a form of intellectual property we’ll explore in more detail later). Such rulings chilled many disclosure initiatives, in- cluding investigations of Philip Morris’s cigarettes and frackers’ chemicals.39

Confi dence in government waned during the stagfl ation of the 1970s, and business lobbyists seized the opportunity to argue that journalists could do a better job at exposing and punishing corpo- rate wrongdoing than bureaucrats. With zealous investigators fer- reting out bad behavior, why bother to require reports? Establish- ment fi gures pooh- poohed complaints that banks were becoming too big, complex, and rapacious. “Sophisticated investors” could un- derstand the risks, they insisted, and banks themselves would avoid duplicity to preserve their reputations.40

Companies tried to maintain an advantage over their competitors by classifying innovative work as “proprietary” or “confi dential.” As computerized exchanges made it possible to gain or lose fortunes within seconds, information advantage became critical throughout the economy. Some economists began to question the wisdom of reg- ulating, or even monitoring, the fast- moving corporate world. Some failed to disclose that they were being paid for “consulting” by the same secretive corporations their writings supported. Business

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

I N T R O D U C T I O N — T H E N E E D T O K N O W 13

schools taught MBAs the basics of game theory, which stressed the importance of gaining an information advantage over rivals.41

Over the last de cade, fortunes made via stealth techniques made secrecy even sexier. Google rose to the top of the tech pack while zealously guarding its “secret sauce”— the complex algorithms it used to rank sites. Investment banks and hedge funds made billions of dollars by courting sellers who didn’t understand the value of what they were holding and buyers who didn’t understand the prob- lems with what they were purchasing.42

While neoliberals were vitiating the regulatory state’s ability to expose (or even understand) rapidly changing business practices, neoconservatives began to advance a wall of secrecy for the deep state.43 In the Nixon administration, Dick Cheney and Donald Rums- feld were already chafi ng at the idea that Congress could force the executive branch to explain its foreign engagements and strategies. When they renewed their executive ser vice in the George W. Bush administration, they expanded the executive branch’s freedom to maneuver (and its power to avoid oversight).44 After 9/11, they pressed even harder for government secrecy, claiming that the only way to win the “war on terror” was for the state to act as clandestinely as its shadowy enemies.45

The Obama administration embraced the expansion of executive secrecy, with far- reaching (and occasionally surreal) results. By 2010, leading intelligence agency experts could not even estimate the over- all costs of the U.S. antiterrorism effort; nor could they map the extent of the surveillance apparatus they had built.46 And their fumbling responses to questions were positively enlightening in comparison with the silence of defense offi cials funded by the “black bud get,” whose appropriations only a sliver of Congress and respon- sible offi cials are privy to understand.47 Big government now stands together with security contractors to manage strategic surprise.

Thus the openness mantra of Progressive Era reformers has been neatly reversed in favor of a Faustian (and credulous) bargain: just keep us safe and we won’t ask about the details. “Nanny state” takes on a very different connotation in this context.

Things weren’t supposed to turn out this way. Little more than a de cade ago, the Internet was promising a new era of transparency,

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

14 T H E B L A C K B O X S O C I E T Y

in which open access to information would result in extraordinary liberty. Law professor Glenn Reynolds predicted that “an army of Davids” would overthrow smug, self- satisfi ed elites. Space physi- cist David Brin believed that new technology would fi nally answer the old Roman challenge, “Who will guard the guardians?” But the powerful actors of business, fi nance, and search did not meekly sub- mit to the fi shbowl vision of mutual surveillance that Brin prophe- sied in The Transparent Society. Instead, they deployed strategies of obfuscation and secrecy to consolidate power and wealth.48 Their opaque technologies are spreading, unmonitored and unregulated.

The Shape of the Book

In this book, I will explore the business practices of leading Internet and fi nance companies, focusing on their use of proprietary reputa- tion, search, and fi nance technologies in our often chaotic informa- tion environment. In some cases, they enable great gains in effi - ciency. In others, however, they undermine both economic growth and individual rights.

The success of individuals, businesses, and their products de- pends heavily on the synthesis of data and perceptions into reputa-

tion. In ever more settings, reputation is determined by secret algo- rithms pro cessing inaccessible data. Few of us appreciate the extent of ambient surveillance, and fewer still have access either to its results— the all- important profi les that control so many aspects of our lives— or to the “facts” on which they are based. Chapter 2 il- lustrates how broadly the new technologies of reputation have infi l- trated society.49

The more we rely on search engines and social networks to fi nd what we want and need, the more infl uence they wield. The power to include, exclude, and rank is the power to ensure that certain pub- lic impressions become permanent, while others remain fl eeting.50 How does Amazon decide which books to prioritize in searches? How does it ferret out fake or purchased reviews? Why do Face- book and Twitter highlight some po liti cal stories or sources at the expense of others?51 Although internet giants say their algorithms are scientifi c and neutral tools, it is very diffi cult to verify those claims.52 And while they have become critical economic infrastruc- ture, trade secrecy law permits managers to hide their methodolo-

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

I N T R O D U C T I O N — T H E N E E D T O K N O W 15

gies, and business practices, defl ecting scrutiny.53 Chapter 3 exam- ines some personal implications of opaque search technology, along with larger issues that it raises in business and law.

Like the reputation and search sectors, the fi nance industry has characterized more and more decisions as computable, programmable procedures. Big data enables complex pattern recognition techniques to analyze massive data sets. Algorithmic methods of reducing judg- ment to a series of steps were supposed to rationalize fi nance, replac- ing self- serving or biased intermediaries with sound decision frame- works. And they did reduce some ineffi ciencies. But they also ended up fi rmly building in some dubious old patterns of credit castes and corporate unaccountability.54 The black boxes of fi nance replaced familiar old problems with a triple whammy of technical complex- ity, real secrecy, and trade secret laws. They contributed to the fi nan- cial crisis of 2008, according to the Financial Times’s John Gapper, because “the opacity and complexity . . . let deception, overpricing and ultimately fraud fl ourish.”55 Perhaps worse, by naturalizing these (avoidable) features of our social landscape, unregulated fi nancial secrecy is starting to give them a patina of inevitability. Chapter 4 examines the role of opaque models and practices in fi nancial markets, along with the challenges they present to citizens, to society, and to the law.

In his book Turing’s Cathedral, George Dyson quipped that “Face- book defi nes who we are, Amazon defi nes what we want, and Google defi nes what we think.”56 We can extend that epigram to include fi - nance, which defi nes what we have (materially, at least), and reputa-

tion, which increasingly defi nes our opportunities. Leaders in each sector aspire to make these decisions without regulation, appeal, or explanation. If they succeed, our fundamental freedoms and oppor- tunities will be outsourced to systems with few discernible values beyond the enrichment of top managers and shareholders.

This book charts two paths of re sis tance. Chapter 5 recommends several legal strategies for checking the worst abuses by black box fi rms. Chapter 6 makes the case for a new politics and economics of reputation, search, and fi nance, based on the ideal of an intelligible society. It would be foolish to hope for immediate traction in today’s gridlocked po liti cal environment. But agencies would need to make “all the right moves” within existing legal frameworks to cabin black

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

16 T H E B L A C K B O X S O C I E T Y

box practices. Moreover, those concerned about the power of Sili- con Valley and Wall Street need to do more than complain about the limited availability of crucial information. We can imagine a future in which the power of algorithmic authority is limited to environ- ments where it can promote fairness, freedom, and rationality.

We do not have to live in a world where hidden scores determine people’s fates, or human manipulations of the stock market remain as inscrutable as the “invisible hand.” We should not have to worry that the fates of individuals, businesses, and even our fi nancial sys- tems are at the mercy of hidden databases, dubious scores, and shad- owy bets. The same technological and legal revolutions that have so far eviscerated personal privacy can be used to protect it and to ad- vance, rather than curtail, our freedoms and our understanding of the social world. Directed at the right targets, data mining and per- vasive surveillance might even prevent the kinds of fi nancial crises and massive misallocations of resources that have devastated the U.S. economy over the past de cade.

We need to promote public values in Internet and fi nance compa- nies, drawing on best practices in other, more regulated sectors. In health care, for example, regulators are deploying technologically savvy contractors to detect and deter fraud, abuse, and unnecessary treatments.57 Similar techniques can and should be applied to keep banks, search engines, and social networks honest.

More transparency would help outside analysts check “irrational exuberance” in markets and uncover corporate misconduct that is now too easily hidden. It might expose unfair competitive or dis- criminatory practices. But as I propose regulatory mea sures, I will repeatedly make the point that transparency is not enough, particu- larly in the fi nance sector. When companies parry with complexity too great to monitor or understand, disclosure becomes an empty gesture. We need to put an end to the recursive games of “disclo- sure” and “tricks to defeat disclosure” that have plagued regulators. Transactions that are too complex to explain to outsiders may well be too complex to be allowed to exist.58

The Self- Preventing Prophecy

We need to face the darker possibilities betokened by current trends. There is a venerable fi ction genre known as the “self- preventing

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

I N T R O D U C T I O N — T H E N E E D T O K N O W 17

prophecy.”59 An author imagines a dystopia, plausibly extrapolating to the future some of the worst trends of the present. If enough readers are shaken from their complacency, they start to make the changes that can prevent the prophecy.60 The author then avoids the fate of Cassandra, the prophetess of Greek myth whose warn- ings were fated to be disregarded. George Orwell’s 1984 and Aldous Huxley’s Brave New World could both be understood in this way, helping to mobilize re sis tance to the totalitarian futures they described.61

Films have also aimed for self- preventing prophecy. In Terry Gil- liam’s Brazil, things start to go downhill for protagonist Sam Lowry after a fl y accidentally jams a printer at an antiterror agency. As he tries to fi x the error, a sclerotic bureaucracy closes in around him, wrongly associating him with violent extremists. Gilliam depicted a state run amok, unaccountable and opaque. Its workings are as mind- less and catatonic as the citizens whom it tortures into submission.62

We like to believe that we have escaped Gilliam’s 1985 dystopia, just as the plausibility of 1984 was eroded by the Eastern Bloc revo- lutions of 1989. Most major decisions about our lives are made in the private sector, not by a state bureaucracy. State- of- the- art com- puters are a far cry from the dusty fi les of the Stasi or the Rube Goldberg contraptions of Gilliam’s imagining.63 The vibrant lead- ers of Wall Street and Silicon Valley are far more polished than the bumbling and brutal beadles of Brazil. Cornucopians urge citizens to simply get out of their way, and to rest assured that technology will solve problems ranging from traffi c jams to freakish weather.

But complacency is unwarranted. Many of these companies make decisions affecting millions of people every day, and small mistakes can cascade into life- changing reclassifi cations. We cannot access critical features of their decision- making pro cesses. The corporate strategists and governmental authorities of the future will deploy their massive resources to keep their one- way mirrors in place; the advantages conferred upon them by Big Data technologies are too great to give up without a fi ght. But black boxes are a signal that information imbalances have gone too far. We have come to rely on the titans of reputation, search, and fi nance to help us make sense of the world; it is time for policymakers to help us make sense of the sensemakers.

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms

18 T H E B L A C K B O X S O C I E T Y

In their workplaces and in their homes, Americans are increas- ingly infl uenced— some might say bullied— by managers who keep their methods under wraps. Corporations depend on automated judgments that may be wrong, biased, or destructive. The black boxes of reputation, search, and fi nance endanger all of us. Faulty data, invalid assumptions, and defective models can’t be corrected when they are hidden. This book exposes them, and proposes solutions.

This content downloaded from �����������13.210.198.208 on Tue, 09 May 2023 07:11:31 +00:00�����������

All use subject to https://about.jstor.org/terms