PADM530Wk7&

Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=fint20

Intelligence and National Security

ISSN: 0268-4527 (Print) 1743-9019 (Online) Journal homepage: https://www.tandfonline.com/loi/fint20

What’s the problem? Frameworks and methods from policy analysis for analyzing complex problems

Stephen Coulthart

To cite this article: Stephen Coulthart (2017) What’s the problem? Frameworks and methods from policy analysis for analyzing complex problems, Intelligence and National Security, 32:5, 636-648, DOI: 10.1080/02684527.2017.1310983

To link to this article: https://doi.org/10.1080/02684527.2017.1310983

Published online: 22 May 2017.

Submit your article to this journal

Article views: 2713

View related articles

View Crossmark data

Citing articles: 5 View citing articles

IntellIgence and natIonal SecurIty, 2017 Vol. 32, no. 5, 636–648 https://doi.org/10.1080/02684527.2017.1310983

ARTICLE

What’s the problem? Frameworks and methods from policy analysis for analyzing complex problems

Stephen Coulthart

ABSTRACT The importance of problem structuring – the activity of making sense of problems – has been grasped by many scholars of policy analysis, a profession that shares much in common in form and function with intelligence analysis. This article imports some of the lessons, frameworks and methodologies of problem structuring to intelligence analysis from policy analysis. The concept of a Type III error is introduced, the analytical mistake of misunderstanding a problem, along with several methodologies designed to help analysts structure problems. One such methodology from policy analysis, called boundary analysis, is demonstrated on a national security case, the 2014 Syrian chemical weapons destruction process.

The world is an increasingly complex place. The portability and exponential improvement in comput- ing power of communication devices, such as smart phones and laptops, allows individuals to create and coordinate better than ever before. As a result, individuals are better able to learn and adapt. Politics play a role as well. The geo-political shift from a bipolar to multipolar world after the Cold War has increased the number of actors on the world stage and global instability. Even changing social structures are fueling complexity as rigid hierarchies give way to more fluid forms of organization best described as networks.

The increasing complexity of problems requires a corresponding focus on the process of making sense of problems – termed ‘problem structuring.’ It is an often ignored but necessary aspect of analysis. It is a cyclical, iterative and often intuitive process undertaken in every field of human inquiry, often unconsciously. Yet, it is the metaphorical compass that guides all inquiry. Failure to properly structure a problem leads to invalid analysis at best, and at worse, contributes to faulty decision-making. This lesson has been grasped by some scholars of policy analysis, an intellectual activity that shares much in form and function with intelligence analysis.

This article highlights the importance of problem structuring and imports the lessons, frameworks and methodologies of problem structuring to intelligence analysis from policy analysis, and its attendant field, policy studies. Much of what follows is based on the research of policy studies scholar William N. Dunn and his foundational research on problem structuring.

Policy and intelligence analysis: similar in form and function

Policy analysis is practiced in nearly every corner of government, from the local to the federal level, in diverse domains such as environmental conservation and national defense, to name a few. Given this diversity, policy analysis ‘does not fit neatly into the disciplinary map.’1 Still, since the 1970s policy

© 2017 Informa uK limited, trading as taylor & Francis group

CONTACT Stephen coulthart [email protected]

INTELLIGENCE AND NATIONAL SECURITY 637

analysis has emerged as a standalone discipline with its own degree programs. Some programs, such as Harvard’s Kennedy School emphasize rigorous quantitative methodology training, whereas others, most notably the Goldman School at University of California, Berkeley, focus on methodological as well as leadership skills.

Despite its application in many domains, scholars have developed useful definitions of policy anal- ysis. Policy studies scholars Weimer and Vinning define it as ‘client-oriented advice relevant to public decisions and informed by social values.’2 The advice-giving function of analysis is emphasized in this definition. Others focus on the analytic process, such as William Dunn’s definition: ‘[policy analysis is] an applied social science discipline which uses multiple methods of inquiry and argument to produce and transform policy-relevant information that may be utilized in political settings to resolve policy problems.’3 This definition recognizes the problem solving aspect of policy analysis and how its results inform decision-making.

Defining intelligence analysis is also difficult because it too is practiced in diverse domains. In the U.S. Intelligence Community (IC) there are approximately 20,000 government analysts excluding a larger number of contractors.4 Analysts working in the IC analyze a wide-array of issues related to international terrorism, nuclear proliferation, geopolitical concerns, and weapons systems, among others. Intelligence analysis is also increasingly conducted in law enforcement and the private sector. Common to all types of intelligence analysis, analysts are trained in a variety of disciplines in the social sciences and science, technology, engineering and math. Similar to the early years of policy analysis, intelligence analysis is undergoing professionalization with initiatives in education and research as evidenced by the growing number of intelligence degree programs offered throughout the U.S. and the world.5 Several journals provide scholars and practitioners venues to publish research and commentary, such as Intelligence and National Security and International Journal of Intelligence and Counterintelligence.

An early description of intelligence pioneer Sherman Kent notes intelligence analysis is ‘[a] natural endeavor to get the sort of knowledge upon which a successful course of action can be rested. And strategic intelligence, we might call the knowledge upon which our nation’s foreign relations, in war and peace, must rest.’6 In his definition, intelligence trainer and scholar Jack Davis echoes Kent on the provision of useful of knowledge to decision makers. He writes that intelligence ‘help[s] policy clients thwart threats and leverage opportunities for US security.’7 Another definition examines the process of intelligence analysis as the activity of ‘evaluating and transforming raw data into descriptions, expla- nations, and conclusions for intelligence consumers.’8

Comparing policy and intelligence analysis definitions points to some important similarities. Both have the same function: supporting decision-making with analysis by producing useful knowledge from raw data. There are a variety of reasons why decision makers need the insight of a policy or intelligence analyst. The most common reason is to hedge against uncertainty. Others may request assistance from analysts to explain a complex trend or idea. Still some decision makers use analysis to justify a course of action they have already decided to take.9 It is worth noting that policy analysts differ from their intelligence counterparts in how they communicate their findings to decision makers.10 Policy analysts are charged with providing explicit policy recommendations to decision makers, however, intelligence analysts are prohibited from making explicit policy prescriptions.11 Instead, intelligence practitioners provide warning and identify potential opportunities for decision makers to exploit.

A second similarity – and most pertinent for this discussion – is that both policy and intelligence analysts are tasked with making sense of complex problems. Defined, problems are phenomena that cause concern or worry for analysts’ customers. Both intelligence and policy analysts make sense of problems through a cyclical, iterative and often intuitive process. There are many names and related concepts for this activity in the policy and intelligence studies literatures, including sensemaking, prob- lem delimitation, and problem definition.12 These related terms are encapsulated in the overarching concept of ‘problem structuring.’ The most explicit model of problem structuring was developed by William Dunn, which encompasses four interdependent activities: problem sensing, problem search, problem definition, and problem specification (see Figure 1).13 While the process is presented formally here, it is often conducted intuitively without explicit recognition of the analyst.

638 S. COULTHART

Analysts are introduced to a problem through a ‘problem situation,’ a vague set of concerns or wor- ries. Since problems are socially constructed and subjective in nature, they are experienced differently, or in some cases not at all, depending on the person doing the sensing. For example, what may be a problem to the Secretary of Defense may not be to the Director of the FBI given their respective mis- sions and roles. Problem situations can be encountered through the analysts themselves or through decision makers. In the case of the latter, decision makers often present analysts with vague problem situations rather than concrete ones; as both policy and intelligence analysts can attest, rarely, if ever, they are provided with a clear problem from a decision-maker.14

Next the analyst constructs a meta-problem composed of many viewpoints or problem representa- tions. Again, this process usually occurs intuitively and reflexively. The subjective nature of problems means that they can be viewed from a number of ways depending on the experience, expertise, and identity of the analyst.15 An example from American politics clarifies the point: while Republicans view mass shootings as the result of a crisis in mental health, Democrats are likely to view it as the result of loose gun control laws. A policy analyst formulating a meta-problem of mass shootings should take into account all of these representations to get the most comprehensive meta-problem.

Using the meta-problem the analyst constructs a substantive problem. This is the re-stating of the problem in its most basic terms. Mass shootings are definable as a legal and psycho-social problem so the problem substantiation is: ‘Mass shootings in the U.S. are the result of a deteriorating mental health system and the wide-scale availability of firearms.’ Similar to the previous phases of problem structuring, the analyst’s perspective plays an important role in defining the substantive problem. Their agency, personal background, and even political views, can shape how they define the problem.16 Analysts at the Department of Justice, for example, are likely to define a terrorism-related problem in legal or criminal terms while those from the Department of Defense will frame it as national security threat.

The final phase is a formal specification of the problem situation using a model, a simplified rep- resentation of reality. Usually, intelligence analysts write reports that serve as verbal models. A narrative is a model because it is a simplification of the reality reported in it. Verbal models have the advantage of being easily communicated but can also contain implicit assumptions in the narrative.17 Formalized models are constructed in two broad types: generic and combination.18 Generic models are templates that can be applied to any problem. One example is a relationship model which illustrates the relation- ships between objects, such as people, places, things, and events. Social network analysis diagrams are a form of a relationship model that is common in intelligence analysis. Combination models, as the name implies, combine multiple models. One type is a geospatial model that implements time and multiple data layers showing events and ethnic groups. Once a model is developed, either formally or informally, an analyst can always return and reformulate it, thus beginning the problem structuring process again.

METAPROBLEM

PROBLEM SITUATION

SUBSTANTIVE PROBLEM

FORMAL PROBLEM

Problem Search Problem Definition

Problem SpecificationProblem Sensing

Figure 1. the phases of problem structuring reproduced from dunn (2012).

INTELLIGENCE AND NATIONAL SECURITY 639

Type III errors: solving the wrong problem

Problem structuring is the most important task policy and intelligence analysts undertake because it guides the rest of the analysis. With an improper understanding of a problem, policy analysts may recommend the wrong solutions and their intelligence counterparts may fail to understand a threat or opportunity of interest to a customer. A simple exercise illustrates this point.

Consider the nine-dot problem.19 Examine Figure 2 and attempt to draw four straight lines without lifting the pencil from the paper. Now look at Figure 3 to see two solutions. Solution (A) assumes that the solver can go outside the box of circles. Interestingly, most solvers unconsciously narrow the prob- lem solution strategies by assuming that they must stay within the square shape formed by the circles therefore making it impossible to solve the problem. Solution (B) takes it one step further and shows that solvers do not need to draw the line through the center of the circles. In both successful solution strategies how the problem is understood (read: structured) is a prerequisite for solving it.

When the solvers narrow the problem definition of the 9-dot exercise by staying inside the square of dots or assuming the lines must pass through the center of the dots, they are committing a type III error.20 Defined it is the error of misunderstanding the problem (An important question then, is how does an analyst know when they have a sufficiently wide enough definition of the problem? We will return to this issue later using the concept of a ‘stopping rule’). The term ‘type III’ is derived from the language of statistics. In hypothesis testing the acceptance of a hypothesis as true when it should have been rejected is called a Type I error, a false alarm as it were. An example of this error occurred in the lead-up to the 1973 Yom Kippur war between Israel and Egypt, Jordan, and Syria. The Egyptian Army mobilized 19 times in the previous 3 months before the actual invasion in October, each time causing Israel to commit a type I error and unnecessarily ready their forces. A type II error is the opposite: a hypothesis is rejected as false when in reality it was true. Screening terrorism watch lists illustrates the problem well: while most suspects do not pose an imminent danger, the analyst must be careful to not miss a suspect bent on taking action.

Figure 2. the nine dot problem.

Figure 3. Solution strategies for the nine-dot problem.

640 S. COULTHART

Type III errors lead to policy and intelligence failures, perhaps more so than Type I and II errors. A classic example is the Vietnam War, which was framed by policy analysts at the Pentagon as a guns and bombs problem. They reasoned that if sufficient power firepower could be projected as it had been in World War II, then victory would follow. Therefore, progress was measured by keeping body counts of North Vietnamese Army and Viet Cong dead. Structuring the problem this way blinded analysts from the political dynamics of winning support for the war at home. Decision makers were surprised by the level of public outcry occurring at home – which ultimately was more pivotal than any North Vietnamese military campaign could have ever been.21

More recently, the authors of the National Intelligence Estimate (NIE) on Iraq’s weapons of mass destruction program committed a type III error. The problem was represented as a technical question of whether Iraq possessed WMDs.22 This problem framing directed analytical resources towards try- ing to physically locate the weapons or information that corroborated their existence. An alternative representation of the problem might have examined additional dimensions of problem, such as the intentions of the Saddam Hussein regime. As Intelligence scholar Gregory Treverton notes, the NIE ‘might have raised the question: Could Saddam be more afraid of his local enemies than he is of the United States? Could that lead him to boast that he had weapons he really didn’t have?’23 Structuring the problem to take into greater account Hussein’s intentions might have helped analysts think beyond the more narrowly defined problem formulation.

The complexity of a problem increases the likelihood of a type III error because the more dynamic the components of it the harder it is to fully understand. A simple problem, such as the 9-dot exercise above, has no dynamics or interacting parts; the positions of the dots are fixed, and therefore, it is not complex. However, most intelligence and policy problems are dynamic, thus making it harder to fully understand the problem. Treverton (2007) identifies a continuum of intelligence problems from least to most complex: puzzles, mysteries, and complexities.24

Puzzles have clear solutions and strategies. During the Cold War, analysts dealt mainly with puzzles: ‘How many missiles did the Soviet Union have? Where were they located? How far could they travel? How accurate were they?’25 As these questions suggest, puzzles have definitive answers and as a result, getting more information is essential to puzzle solving and avoidance of a type I or II error. If an analyst needs to estimate the number of North Korean warheads, more satellite imagery and human intelli- gence reports are going to get him closer to the answer. At the same time, the well-structured nature of puzzles make type III errors less likely.

Unlike puzzles, ‘mysteries’ have no definitive answer because the outcomes are contingent on other events. For example, a puzzle might require an analyst to estimate the number of nuclear missiles North Korean’s currently possess and a mystery would be to guess how many they will have in twenty years. There is simply no definitive answer to the latter question at this moment. As a result, analysts can never truly confirm if they are right or wrong. Fortunately, mysteries have previous experience and/or theory that can assist in structuring the problem. For example, analysts estimating North Korea’s number of nuclear weapons cache have previous decades to analyze and project into the future.

The most difficult problems in intelligence are ‘complexities.’26 Treverton notes that complexities are ‘mystery-plus.’ Unlike mysteries, pure complexities have little or no structure; there is no past to draw previous experience or theory to assist analysts in structuring the problem. A possible outcome of a complexity is a ‘black swan,’ a low probability, high impact event.27 However even the most complex of complexities have some basis to structure problems, if the analyst is savvy enough to find the signal in the noise. The case of the September 11 attacks is instructive. Previous experience suggested that some Islamist extremist groups had considered weaponizing airliners: during the 1990s the Armed Islamic Group considered crashing an airliner into the Eiffel Tower.28 Some researchers took notice of this fact. Brian Jenkins noted in 1989, ‘the nightmare of governments is that suicidal terrorists will hijack a commercial airliner and, by killing or replacing its crew, crash into a city or some vital facility.’29 This example illustrates that in a world of complexities Type III errors may be a more likely threat to intelligence than Type I or Type II errors.

INTELLIGENCE AND NATIONAL SECURITY 641

Avoiding type III errors: selecting problem structuring methods

As noted in the introduction, the complexity of problems is increasing and as a result analysts are more likely to commit type III errors. How then should analysts effectively structure complex problems? Of course, problem structuring can be conducted intuitively. This approach is faster but it also lacks replicability and transparency, a problem in contexts where there is attention paid to accountability.

The intelligence analysis literature contains hundreds of analytical methods, although but provides less guidance on when to select specific methods for structuring problems. Fortunately, the policy analysis literature provides some guidance on selecting problem structuring methodologies through the principle of methodological congruence.30 It asserts that there are two types of methods, first and second-order. Understanding the difference between the two is important because using a first-order method on an unstructured problem increases the probability of a type III error.

First-order methods are useful for solving structured problems, that is, problems that have already been structured by an analyst. A good way to determine if a method is first-order is to consider if the purpose of the method is to come to a conclusion about an analytical problem, such as a prediction or judgment. Most of the puzzle-type problems listed above are immediately applicable to be solved with first-order methods without problem structuring. A common first-order method in intelligence is analysis of competing hypotheses (ACH). To use ACH, analysts fill out a matrix with hypotheses in the columns and evidence in the rows. Then each piece of evidence is judged by how consistent it is with each hypothesis. After evaluating the evidence the least disconfirmed hypothesis is considered the most plausible. In ACH, the analyst reaches a conclusion in the form of the least falsified hypothesis.

Second-order methods are problem structuring methods that help analysts structure the problem. That is, these methods are used to understand the problem rather than drawing inferences about it. Most mysteries and complexities require the use of second-order methods to understand the problem before application of a first-order method. To assist analysts in selecting problem structuring methodol- ogies, there are two types of second-order methods, which I term data-driven and judgment-driven. The former utilize digitized data as inputs and the latter use information inputs from analysts’ judgments.

Data-driven methods require access to digitized data and a software program to conduct the anal- ysis. While many of these methods were too expensive two decades ago, advances in computing and digital storage have increased their utility. Content analysis, the examination of texts to find patterns and meanings, is illustrative.31 The method is often used manually but the careful examination of texts is very time consuming; a careful analysis of dozens of pages requires many hours. However, modern content analysis software programs can accomplish the process in seconds. These software programs dredge documents for patterns, such as the frequency or relation of concepts, based on presets inputted into a software program.

The development of a criminal profile with data-driven problem structuring methods illustrates how these methods are used in practice. Since offender data is usually digitized, criminal intelligence analysts can use data to develop criminal profiles. One study explained how data mined from three datasets from police in London – persons arrested by the police, drug seizures, and police seizures – was used to build a profile of heroin drug dealers.32 According to the analysis, the average heroin dealer is female, Afro-Caribbean, aged 25–35 from one of three neighborhoods in London. The profile, a type of model, also provides insights into where dealers are typically caught, whether they resist arrest violently, and the mode of sale, among other characteristics.

A strength of the data-driven problem structuring methods is their replicability, the ability of other analysts to reproduce the findings (or even the same analyst). This trait is important for analysis because replication increases the credibility and defensibility of analytic conclusions to decision makers.33 Since data-driven methods require that analysts use software programs with inputs (e.g. the words the com- puter is searching for) it increases the transparency and, therefore, the replicability of the analysis.

Despite the promise of data-driven problem structuring methods, there are limitations. First, it is sometimes difficult to find analysts with the skills to use these sophisticated methodologies. One com- mentator in the IC notes that while many analysts have extensive substantive experience in regional

642 S. COULTHART

and/or functional areas, they lack skills to collect, manage, and analyze large datasets using modern data-driven methods.34 Second, the strength of data-driven methodologies is also a weakness because the automation common in data-driven methodologies requires close supervision to ensure that the computer is finding what the analyst is looking for. This is particularly important in cases where meaning and context is nuanced.35

Judgement-driven methods are a less sophisticated, but an efficient option. Instead of relying on collected data, these methods use the judgments of analysts.36 Many of these methods are called ‘struc- tured analytic techniques’ (SATs) by intelligence analysts, although not all SATs are useful for problem structuring as the discussion of ACH as a first-order method illustrates. SATs have received increased attention most notably through the Intelligence Reform and Terrorism Prevention Act (2004) which mandates analysts receive training in them.37 As of 2011, at least 4,000 analysts had received training in SATs through a training program called ‘Analysis 101’ representing almost a quarter of IC’s analytic cadre.38 SATs are considered structured because they provide an ‘audit trail’ for others to determine the process by which the analyst reached their conclusions.39 Another defining characteristic of SATs is that they are ideally used collaboratively to generate more ideas and information into the analytical process.40

Two common judgment-driven problem structuring methods are brainstorming and scenario plan- ning. Brainstorming was developed by advertising executive Alex Osborn to help groups come up with more ideas. An underlying assumption of the method is that structuring the deliberation process and suppressing criticism of ideas will increase a group’s creativity.41 There are variations on how to use the method, but it typically involves a facilitator collecting ideas through a flipchart or sticky notes from all participants. Afterward the group sorts and prioritizes the most important ideas. An application of brainstorming for problem structuring might include identifying all of the possible trafficking routes of heroin across the U.S.-Mexican border to create a map of the threats. Scenario planning (also known as alternative futures analysis) requires analysts to identify important drivers that lead to different out- comes. Analysts then construct scenarios based on different levels of drivers.42 For example, the cost of narcotics and level of cartel competition are important drivers affecting the level of drug-related violence. Analysts can consider how different levels of these drivers (high/low cost and high/low com- petition, respectively) lead to alternative future scenarios.

Judgment-driven methods are easily taught and applied. While it may take many years to master one of these methods, most analysts grasp the basics after a few hours of training and some practice exercises. Judgment-driven methods also have the advantage of requiring little or no data. This is impor- tant because developing events may lack data, rendering data-driven methods less useful. However, a major weakness of judgment-driven methods is that they are notoriously unreliable. This is due to the fact that these methods require analysts to make individual judgments about uncertain events on which they disagree. For example, analysts using scenario planning may disagree over (1) what drivers should be included and (2) how different levels of the variable may effect different projected futures.

A problem that both data and judgment-driven methods have is that they lack a ‘stopping rule,’ a mechanism to tell the analyst that they have collected most, if not all, problem representations. The stop- ping rule is important because it lets the analyst know how likely it is that they have committed a type III error. However, when structuring complex problems there ‘are no criteria for sufficient understanding … because there are no ends to the causal chains that link interacting open systems.’43 In simpler terms, complex problems make it difficult for the analyst to know when to stop using second-order methods. Ideally, problem structuring methods should have a stopping rule built into them to give analysts an indication that they have included enough information to stop trying to structure the problem.

The next section introduces a second-order method from policy analysis called boundary analysis. Developed by policy studies scholar William Dunn, it is used to determine the analytic ‘boundaries’ of complex problems.44 The method addresses the stopping rule problem as well as replicability issues of judgment-driven methods. An example analyzing the Syrian chemical weapons agreement demon- strates how it can be used in intelligence analysis.

INTELLIGENCE AND NATIONAL SECURITY 643

Problem structuring the syrian chemical weapons agreement

In August, 2013, the Assad regime launched rockets tipped with nerve gas into an opposition-held neighborhood in Damascus, Syria’s capital. Casualty estimates vary significantly depending on the source, but range from the US’s assessment of approximately 1500 to a French assessment of 281 killed.45 In early September, as the US was weighing military options, the Syrian government signaled it was will- ing to seek diplomatic solutions to the crisis. From these early talks, a framework emerged for eliminating Syria’s chemical weapons stockpile by becoming a signatory of the Chemical Weapons Convention and working with the Organisation for the Prohibition of Chemical Weapons. In late September, United Nations Security Council passed Resolution 2118 setting a target of mid-2014 for the removal and destruction of Syria’s chemical munitions.

Determining the extent to which the agreement process would be completed was an important and complex problem. Governments were keen to know if the Syrian government would follow through with its promises because failure to destroy the weapons could lead to a deepening international crisis. It was also a complex problem, somewhere between a mystery and a complexity in Treverton’s problem spectrum because the outcome was highly contingent on other events and there was also little previous history of large-scale weapons disposal during an active conflict. In other words, it was exactly the type of complex problem intelligence analysts must regularly grapple with.

The first step in boundary analysis is to sample diverse data sources that contain problem representa- tions. Recall that problem representations are different ways that a problem situation is experienced. To gather diverse representations this case study relied on the viewpoints of 21 foreign affairs experts and 21 graduate students. Respondents were identified through a purposive sampling strategy to identify those with sufficient knowledge of the case. Experts possessed expertise in the Middle East, chemical weapons, civil wars, or a combination of all three. Three intelligence analysts from the IC were included in the sample of experts. Most of the 21 graduate students had coursework in international affairs with an emphasis on security and intelligence studies. While individuals were surveyed for this case study, analysts can also use other sources, such as newspapers and reports to identify problem representations. Respondents were asked to provide subjective probabilities regarding the comple- tion of the destruction process and rationales to support their answers. The rationales were coded for problem representations.

As an analyst codes and compiles the problem representations they will notice an initial exponen- tial increase and then leveling off of them after the 15th to 20th respondent or document, depending on what data is used. After this rapid rise in the number of representations, there will be a very small number of new ones. This is the ‘stopping rule’ of boundary analysis and it is explained by Bradford’s Law, an empirical regularity that states that after searching a few key sources, the analyst will have attained nearly all unique problem representations.46 The leveling-off of problem representations lets the analyst know he has reached a nearly complete set and can cease trying to bound the problem. There also appears to be a temporal dynamic related to the number of problem representations gen- erated. Rickabaugh and Coulthart conducted a boundary analysis using open source data of a criminal intelligence problem and they found that two weeks after the beginning of the crime, nearly all problem representations were elicited.47 It should be noted that analysts can keep searching for new problem representations if time and resources allow, especially with the most complex problems.

The stopping rule in this case study was also found after an initial increase in problem representa- tions. The curved line in Figure 4 represents the expected distribution of cumulative unique hypotheses based on Bradford’s law and the squiggly line the observed distribution. The fit is nearly perfect, equaling a 96 percent match between the expected cumulative distribution predicted by Bradford’s law and the observed results (R2 = .96). In total, 10 distinct problem representations were identified. Combined, these representations constitute the meta-problem that can be developed into a substantive problem.

According to the results of the boundary analysis, the implementation of the chemical weapons agreement is definable by three main problem substantiations: the enforceability of the agreement, Syrian intentions, and the dynamics of the civil war in the region. We may then formally state the

644 S. COULTHART

problem situation as the following: The implementation of the chemical weapons agreement rests on the enforceability of the agreement, regime intentions, and the dynamics of the civil war. These were drawn from the 10 problem representations elicited from the respondents (see Figure 5), some of which fit into more than one problem substantiation. Many of the representations viewed the implementation of the deal through the lens of its enforceability in terms of political, technical, and logistical dimen- sions. Respondents noted that the international resolve to follow through with the agreement would be instrumental. In particular, Russia was cited as potential obstacle to completing the destruction process. Several respondents noted that there simply was not enough time to complete the destruction process, even under ideal conditions.

The second problem substantiation focused on the importance of the Syrian government’s inten- tions. Most of the experts were suspicious of Syrian President Bashir Assad. One went as far as to note ‘people who drop [explosives] on their own folks are not going to willingly give up their weapons.’48 Still, others noted the ease with which the Syrian government could smuggle weapons out of the country before inspectors could collect them. Most respondents identified the general lack of transparency of the agreement, either because of the fog of war or due to sabotage by the Syrian government.

The final substantiation focused on the dynamics of the civil war and how it could obstruct the transportation and destruction of the chemical weapons. One problem representation related to the instability of countries around Syria, specifically through the Sunni-Shia conflict. An expert hypothesized

C um

ul at

iv e

Pr ob

le m

R ep

re se

nt at

io ns

Expert Rank

Figure 4. cumulative problem representations of Syrian chemical weapons agreement implementation.

# of

T im

es M

en ti

on ed

Problem Representations

Figure 5. Problem representations.

INTELLIGENCE AND NATIONAL SECURITY 645

the conflict might cause some of the weapons to ‘end up in caches in Iraq or Iran.’49 Then there were the significant logistical difficulties of finding and transporting the weapons in an active conflict zone. Several respondents noted that this would be a major problem given the ongoing war. In particular, the various armed groups fighting for control were noted. Interestingly, a couple of respondents iden- tified ISIL as a potential threat to the process, months before the group gained notoriety in the West.

Using the problem substantiations, several types of models can be constructed. A simple way to present the information would be to describe it in written narrative as it is reported above. Another option is to represent the problem through a relevance tree model.50 This model is useful for showing the hierarchy of concepts defining the problem. At the highest level is the meta-problem: the imple- mentation of the chemical weapons destruction process (see Figure 6). Moving to a less abstract level, the problem is defined by the three problem substantiations discussed above: (1) the enforceability of the agreement; (2) the Syrian government’s intentions; and (3) the dynamics of the civil war. At the lowest level of the hierarchy are individual issues drawn from the respondent’s problem representations. Analysts could use this model to provide a concise snapshot to a decision-maker of the main issues related to the chemical weapons destruction process. More sophisticated models could be informed by the results of the boundary analysis, such as a Geographic Information Systems (GIS) model. Analysts could create data layers on where rebel groups control territory, the location of Syrian government forces, chemical weapons depots, and any other number of important information surfaced by the problem structuring process.

With this problem structured, first-order (problem solving) methods might be applied, such as ACH. If a decision-maker was interested in the potential of Syrian government meddling in the destruction process (e.g. through smuggling weapons out of the country before inspectors could collect them), the method could be applied to test the hypothesis. More sophisticated first-order methods could be applied, such as game theory to understand how the key players, such as the rebel groups, Russians, and the Assad regime, would respond to various incentives. Regardless of the problem solving method used, the analyst can proceed based on the boundary analysis with the assurance he has reduced the likelihood of misunderstanding the problem (type III error), because the resulting distribution of problem representations conform to Bradford’s law.

Given proper training, boundary analysis is a time efficient and reliable method of structuring com- plex analytical problems. In the author’s experience a pair of analysts with familiarity of the method can structure a complex problem in a couple of working days. This includes time to identify experts, poll them, code and synthesize their responses, and build a model. In the case of the latter, a more compli- cated model will require more time. The method can also be implemented reliably. During the coding phase analysts randomly select documents or responses from experts to save time and code each and

Figure 6. relevance tree model of the Syrian chemical weapons destruction process.

646 S. COULTHART

compare them to determine where they agree and disagree. In most cases any discrepancies can be worked out and it is possible with minimal training to learn how to calculate simple agreement scores between analysts, such as Cohen’s kappa. While these agreement scores are not common in intelligence analysis, they are a helpful way to determine the reliability, and therefore, the replicability of the analysis.

Conclusion

Professor of urban planning Donald Schön notes that ‘in the varied topography of professional practice, there is a high, hard ground where practitioners can make effective use of research-based theory and technique, and there is swampy lowland where situations are confusing ‘messes’ incapable of solution.’51 This article has argued that an area where practitioners are often confined to the ‘swampy lowland’ is intelligence and policy analysis, and that lessons, frameworks, and methods from the latter can assist analysts in problem structuring. The consequences of improperly structuring a problem can at best lead to inaccurate analysis, and at worse, policy disasters. Different types of problem structuring methodol- ogies provide an avenue to avoid Type III error. However, each has its own strengths and weaknesses and so analysts will need to carefully weigh the constraints of the analytic task and available resources, otherwise they run the risk of making the most common, and arguably worst, mistake in intelligence and policy analysis: solving the wrong problem.

Notes 1. Wildavsky, Speaking Truth to Power, 384. 2. Weimer and Vining, “Policy Analysis: Concepts and Practice.” 3. Dunn, Public Policy Analysis, ix. 4. Priest and Arkin, Top Secret America. 5. Marrin and Clemente, “Improving Intelligence Analysis”; For coverage and discussion of U.S. programs see: Coulthart

and Crosston, “Terra Incognita.” For an international view, see: Landon-Murray, “Building an International and Comparative View.”

6. Kent, Strategic Intelligence, viii. 7. Davis, “Leadership in Intelligence Analysis,” 2. 8. Berkowitz and Goodman, Strategic Intelligence, 85. 9. Meltsner, Policy Analysts, 5. 10. There are, of course, other differences between policy and intelligence analysis. The latter is practiced in a secretive

context, which has implications for the analytical process because adversaries are always trying to stay one step ahead. As a result, intelligence analysts deal with more deceptive data than their policy counterparts. See: Johnston, “Developing a Taxonomy,” 62.

11. McLaughlin, “Serving the National Policy-maker,” 71–90. 12. Pirolli and Card, “The Sensemaking Process,” 2–4; Moore Sensemaking; and Veselý, “Problem Delimitation,” 80–100. 13. Dunn, Policy Analysis, 76. 14. Former intelligence analyst Frank Marsh provides an anecdote of this experience: he was tasked a decision-maker

to analyze ‘the heroin threat in the United States,’ an incredibly vague and complex task. Marsh, “The Power of Questions.”

15. Assumptions can shape how mental models are constructed. For example, see: Heuer, Psychology of Intelligence Analysis, 6.

16. A good example of how political beliefs play a role in intelligence analysis is discussed in: Muller, “Intelligence in Red and Blue,” 1–12.

17. Dunn, Policy Analysis, 82. 18. Clark, Intelligence Analysis, 60–86. 19. Adams, Conceptual Blockbusting, 16, 17. 20. Mitroff and Featheringham, “On Systemic Problem Solving.” 21. DeLeon, Advice and Consent, 65. 22. Clark (2010) also identified this as an instance of defining the analytic problem wrong. Clark, Intelligence Analysis,

315, 316. 23. Treverton, “Risks and Riddles.” 24. Treverton, Addressing Complexities. 25. Treverton, “Risks and Riddles.” 26. Treverton, Addressing Complexities, 5–11.

INTELLIGENCE AND NATIONAL SECURITY 647

27. Taleb, The Black Swan. 28. Treverton, “Risks and Riddles.” 29. Quoted in Treverton, Addressing Complexities, 11. 30. Dunn, “Methods of the Second Type,” 723–6. 31. Prunckun, Handbook of Scientific Methods, 61–8. 32. Buscema, “Artificial Adaptive System,” 481–511. 33. Prunckun, Handbook of Scientific Methods, 14. 34. Conway, “Data Science,” 26. 35. For examples of the methodological difficulties of using automated data extraction on national security subject

matter, see: Kenney and Coulthart, “The Methodological Challenges.” 36. The philosophical justification for what I refer to as ‘judge-driven analytical methods’ was first put forward by

Helmer and Rescher, “On The Epistemology,” 25–52. 37. US Congress, “Terrorism Prevention Act.” 38. Defense Intelligence Agency, “Graduating 4,000 Analysts.” 39. Marrin, Improving Intelligence Analysis, 31. 40. Heuer and Pherson, Structured Analytic Techniques, 6. 41. Osborn, Applied Imagination. 42. Schwartz, The Art of the Long View. 43. Rittel and Webber, “Dilemmas in a General Theory,” 162. 44. Dunn, “Methods of the Second Type,” 720–37. 45. Nikitin et al., “Syria’s Chemical Weapons,” 15. 46. Brookes, “Bradford’s Law.” 47. Rickabaugh and Coulthart, “[Bracketing] the Black Swan.” 48. Expert Respondent 3016, Email survey, April 3rd, 2014. 49. Expert Respondent 3002, Email survey, March 31st, 2014. 50. Clark, Intelligence Analysis, 69. 51. Schön, The Reflective Practitioner, 41.

Disclosure statement No potential conflict of interest was reported by the author.

Notes on contributor Stephen Coulthart is an assistant professor of Security Studies in the National Security Studies Institute at the University of Texas at El Paso. His research focuses on intelligence analysis and illicit networks, such as terrorist and criminal groups. Professor Coulthart has had research published in Intelligence and National Security, the International Journal of Intelligence and Counterintelligence, and the Journal of Conflict Resolution, among others. He holds a PhD in Public and International Affairs from the University of Pittsburgh as well as two master’s degrees in international relations and public administration, both from Seton Hall University. While at the University of Pittsburgh, he received the Harold D. Lasswell award from the Horowitz Foundation for his dissertation research on analytic reform in the U.S. intelligence community.

Bibliography Adams, James. Conceptual Blockbusting: A Guide to Better Ideas. New York: W.W. Norton & Company, 1976. Berkowitz, Bruce D., and Allan E. Goodman. Strategic Intelligence for American National Security. Princeton, NJ: Princeton

University Press, 1991. Brookes, Bertram. “Bradford’s Law and the Bibliography of Science.” Nature 224, no. 5223 (1969): 953–956. Buscema, Massimo. “Artificial Adaptive System for Parallel Querying of Multiple Databases.” In Intelligent Data Mining in

Law Enforcement Analytics, edited by Paolo Massimo Buscema and William J. Tastle, 481–511. New York: Springer, 2013. Clark, Robert M. Intelligence Analysis: A Target-centric Approach. Thousand Oaks, CA: CQ press, 2010. Coulthart, Stephen, and Matthew Crosston. “Terra Incognita: Mapping American Intelligence Education Curriculum.” Journal

of Strategic Security 8, no. 3 (2015): 46–68. Conway, Drew. “Data Science in the U.S. Intelligence Community.” IQT Quarterly 2, no. 4 (2011): 24–27. Davis, Jack. “Leadership in Intelligence Analysis.” Unpublished manuscript. August, 2002. Defense Intelligence Agency. “Graduating 4,000 Analysts from Analysis 101.” December, 2011. DeLeon, Peter. Advice and Consent: The Development of the Policy Sciences. New York: Russell Sage Foundation, 1989.

648 S. COULTHART

Dunn, William N. “Methods of the Second Type: Coping with the Wilderness of Conventional Policy Analysis.” Review of Policy Research 7, no. 4 (1988): 720–737.

Dunn, William N. Policy Analysis. 1st ed. Englewood Cliffs, NJ: Prentice Hall, 1981. Dunn, William N. Policy Analysis. 5th ed. Upper Saddle River, NJ: Pearson, 2012. Helmer, Olaf, and Nicholas Rescher. “On the Epistemology of the Inexact Sciences.” Management Science 6, no. 1 (1959):

25–52. Heuer, Richards J. Psychology of Intelligence Analysis. Washington, DC: Center for the Study of Intelligence, 1999. Heuer, Richards J., and Randolph H. Pherson. Structured Analytic Techniques for Intelligence Analysis. Washington, DC: CQ

Press, 2010. Johnston, Rob. “Developing a Taxonomy of Intelligence Analysis Variables.” Studies in Intelligence 47, no. 3 (2002): 61–71. Kenney, Michael, and Stephen Coulthart. “The Methodological Challenges of Extracting Dark Networks: Minimizing False

Positives through Ethnography.” In Illuminating Dark Networks: The Study of Clandestine Groups and Organizations, edited by Luke Gerdes, 52–70. New York: Cambridge University Press, 2015.

Kent, Sherman. Strategic Intelligence for American World Policy. Princeton, NJ: Princeton University Press, 1949. Landon-Murray, Michael. “Building an International and Comparative View of Academic Intelligence Education.” American

Intelligence Journal 31, no. 2 (2013): 12–22. Marrin, Stephen. Improving Intelligence Analysis: Bridging the Gap between Scholarship and Practice. New York: Routledge,

2012. Marrin, Stephen, and Jonathan D. Clemente. “Improving Intelligence Analysis by Looking to the Medical Profession.”

International Journal of Intelligence and CounterIntelligence 18, no. 4 (2005): 707–729. Marsh, Frank. “The Power of Questions.” Presented at the 10th Annual International Association for Intelligence Education

Conference, Erie, PA, June 2014. McLaughlin, John. “Serving the National Policymaker.” In Analyzing Intelligence: Origins, Obstacles, and Innovations, edited

by Roger Z. George and James Bruce, 71–90. Washington, DC: Georgetown University Press, 2008. Meltsner, Arnold J. Policy Analysts in the Bureaucracy. Los Angeles: University of California Press, 1976. Mitroff, Ian I., and Tom R. Featheringham. “On Systemic Problem Solving and the Error of the Third Kind.” Behavioral Science

19, no. 6 (1974): 383–393. Moore, David T. Sensemaking: A Structure for an Intelligence Revolution. Washington, DC: National Intelligence University

Press, 2012. Muller, David. “Intelligence in Red and Blue.” International Journal of Intelligence and CounterIntelligence 21, no. 1 (2008): 1–12. Nikitin, Mary Beth D., Paul K. Kerr, and Andrew Feickert. Syria’s Chemical Weapons: Issues for Congress. Washington, DC:

Congressional Research Service, 2013. Osborn, Alex F. Applied Imagination, Principles and Procedures of Creative Thinking. New York: Charles Scribner’s Sons, 1953. Pirolli, Peter, and Stuart Card. “The Sensemaking Process and Leverage Points for Analyst Technology as Identified through

Cognitive Task Analysis.” In Proceedings of International Conference on Intelligence Analysis 5 (2005): 2–4. Priest, Dana, and William M. Arkin. Top Secret America: The Rise of the New American Security State. New York: Little, Brown,

2011. Prunckun, Hank. Handbook of Scientific Methods of Inquiry for Intelligence Analysis. Lanham, MD: Rowman and Littlefield, 2015. Rittel, Horst W. J., and Melvin M. Webber. “Dilemmas in a General Theory of Planning.” Policy Sciences 4, no. 2 (1973): 155–169. Schön, Donald A. The Reflective Practitioner: How Professionals Think in Action. New York: Basic books, 1983. Schwartz, Peter. The Art of the Long View: Paths to Strategic Insight for Yourself and Your Company. New York: Broadway

Business, 1996. Taleb, Nassim Nicholas. The Black Swan. New York: Random House, 2010. Treverton, Gregory F. Addressing Complexities in Homeland Security. Sweden: The Swedish National Defence College, 2008. Treverton, Gregory F. “Risks and Riddles.” Smithsonian Magazine, June 2007. Veselý, Arnošt. “Problem Delimitation in Public Policy Analysis.” Central European Journal of Public Policy 1, no. 1 (2007):

80–100. Weimer, David L., and Aidan R. Vining. Policy Analysis: Concepts and Practice. 4th ed. Upper Saddle River, NJ: Prentice, 2005. Wildavsky, Aaron B. Speaking Truth to Power. New Brunswick, NJ: Transaction Publishers, 1989. US Congress. “Terrorism Prevention Act of 2004.” Public Law 458 (2005).