RESERVED FOR HIFSA NO ONE ELSE REPLY

Using the Living Laboratory Framework as a Basis for Understanding Next Generation Analyst Work

Michael D. McNeese, Vincent Mancuso, Nathan McNeese, Tristan Endsley, Pete Forster

The Pennsylvania State University, College of Information Sciences and Technology, IST Building, University Park, PA 16803

ABSTRACT

The preparation of next generation analyst work requires alternative levels of understanding and new methodological departures from the way current work transpires. Current work practices typically do not provide a comprehensive approach that emphasizes the role of and interplay between (a) cognition, (b) emergent activities in a shared situated context, and (c) collaborative teamwork. In turn, effective and efficient problem solving fails to take place, and practice is often composed of piecemeal, techno-centric tools that isolate analysts by providing rigid, limited levels of understanding of situation awareness. This coupled with the fact that many analyst activities are classified produces a challenging situation for researching such phenomena and designing and evaluating systems to support analyst cognition and teamwork. Through our work with cyber, image, and intelligence analysts we have realized that there is more required of researchers to study human-centered designs to provide for analyst’s needs in a timely fashion.

This paper identifies and describes how The Living Laboratory Framework can be utilized as a means to develop a comprehensive, human-centric, and problem-focused approach to next generation analyst work, design, and training. We explain how the framework is utilized for specific cases in various applied settings (e.g., crisis management analysis, image analysis, and cyber analysis) to demonstrate its value and power in addressing an area of utmost importance to our national security. Attributes of analyst work settings are delineated to suggest potential design affordances that could help improve cognitive activities and awareness. Finally, the paper puts forth a research agenda for the use of the framework for future work that will move the analyst profession in a viable manner to address the concerns identified.

Keywords: cognitive systems engineering, cognitive science, learning, analyst work, knowledge elicitation, simulation

1. INTRODUCTION Generally speaking, work involves regular routines that are familiar and well-rehearsed sequences of activities that accomplish intentions. However, in reality, this definition might be much too simplistic as work also involves many new emerging situations that require responding to various levels of complexities. Complexities arise from many sources and can a) produce adaptation to change, b) result in anxiety and stress, c) consume much attention especially if errors and unintended consequences are present, and d) create the need to overcome previous learning (interference effects). A seminal example of where complex work may occur is in the job of information analysts. In many domains, analyst work is often ill-defined, unstructured, dynamic, uncertain, and emerges over time in unexpected ways. It involves cognition (human thought), computation (technology), context (environments), collaboration (working together) and culture (differences), all orchestrated according to demands that arise and need to be addressed. Challenges brought forth through complex work result in it being difficult to understand, hard to process while maintaining focus, and producing an inherent nature of ambiguous decision-making, often leading to second- guessing and re-planning. Complex analyst work may be unpredictable and surprising [1] and even violate expectations when interdependent activities are out of synch or in opposition. On the other hand, there are also more routine, prescribed elements that can be learned and practiced (e.g., an intelligent analyst might use specific tools to link together various events within a given timeframe and traces actors and their locations within an event as part of their best practices). In spite of the threat of conflicts, constraints, and possible disorder, complex analyst work often produces success dependent on how success is defined and measured. Much of our own research in understanding and improving complex analyst work has been approached through the use of cognitive systems engineering methods [2, 3] and naturalistic decision making [4] frameworks. Historically, the use of specific techniques, such as concept mapping, IDEF

Next-Generation Analyst, edited by Barbara D. Broome, David L. Hall, James Llinas, Proc. of SPIE Vol. 8758, 87580F · © 2013 SPIE · CCC code: 0277-786X/13/$18

doi: 10.1117/12.2016514

Proc. of SPIE Vol. 8758 87580F-1

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

structural decompositions, and design storyboards helped to represent integrative components of work within a specified context. The need for human-system integration within complex analyst work is important and these initial applications help to discern and define many coupled problems. Building from this approach, research recognized the need for a more holistic, interdisciplinary perspective, rather than an isolated approach producing piecemeal analyses. Hence, the Living Laboratory Perspective began [5]. The objective of this paper is to articulate the Living Lab in application to next generation analyst work. Based on work that has been conducted in several domains with differing contexts, we present an overview of our work and experiences, as well as a forecast for future research. 1.1 Complex Work in Warfare

Within warfare, complex analyst work is often required: it is subject to many dynamic variances that can increase stress, pressure, and emotions to physical and mental breaking points. Yet, many brave men and women have accomplished incredible acts to the point of valor. Warfare may produce both success and failure, with a variety of consequences between these polarities. It is transmitted across many contexts, and situations, all of which may impose unique constraints, delays in progress, action potentials, and dilemmas dependent upon time-space-distance tradeoffs. The following [6] description identifies what may be expected when warfare becomes complex work:

“Warfare is often defined as acts of opposition between competing forces. Opposition typically involves the weakening or breakdown in the opposing force’s power and/or will. Strategies and tactics are introduced to facilitate breakdowns sooner. Strategies and tactics involve highly complex missions consisting of weapons, support systems, and various resources all of which require effective logistical …. Mission plans change as a function of (1) threats induced by the enemy (2) what is known / not known about opposing forces (i.e., intelligence) (3) consideration of environmental instabilities (e.g., terrain, weather, night operations) (4) remaining resources and supplies; to mention just a few of the factors that need to be traded-off as solutions are imparted. Warfare, however, is highly enabled (and constrained) by humans working together at various levels of an operation. Humans form teams, which in turn form battalions. Teamwork is everywhere as battle management ensues and as other teams carry out plans in situ. This is the nature of the warfighters and their field of practice. [McNeese, 1998, p. 161]”

Warfare engulfs humans in many situations, problems, and decisions that can dramatically affect the future. Humans have always used different kinds of tools and technologies to gain tactical and strategic advantage against threats. Moreover, humans have worked together to carry out missions with those tools in challenging settings in order to perform to their highest capacity. Within this vision of success, many challenges can be present that thwart success and become barriers to achieving desired outcomes. At this point the paper will focus specifically on addressing these challenges through the lens of analysts working together to gain superiority in a mission.

2. COMPLEX ANALYST WORK Over the past 25 years, research in warfare has often focused on traditional analyst’s work in prototypical command centers that involve face-to-face, collocated synchronous operations. More recent research (10 years to present) articulates the change in analyst’s work, which is now often highly distributed through synchronous and asynchronous means. From the traditional analyst perspective (e.g., the intelligent analyst), we consider work in a stationary command post, using typical reports of intelligence sources and tools to extrapolate and plot information (e.g., the Analyst’s Notebook). However, as technology continues to evolve, so must analysts. Analysts often need to utilize seamless mobile tools that employ techniques such as crowd sourcing, video integration, big data synopsis, social media visualization tools, and knowledge elicitation probes to accomplish their job.

Within this paper we discuss complex analyst work across three distinct contexts: (1) C3I Battle Management1 Work, (2) Intelligence / Image Analyst Work, (3) Cyber-Security Work [7, 8, 9, 10]. Unlike the traditional perspective of analyst work, each of these three domains evolves within socio-technical systems containing interaction between information, technology, people and cognition (ITP-C). While similar at a high level, each of these domains has unique characteristics in their specific work, which in-turn has an effect on how one designs interfaces, tools, and aids for that given setting. With these inherent differences, the traditional, brittle techniques are not suited to study these domains. Rather, we have

1 Note that much of our work from crisis management/crisis response is similar to that of battle management.

Proc. of SPIE Vol. 8758 87580F-2

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

applied the Living Lab, which affords more domain flexibility, to build a complete understanding of the cognitive and technological work, and how to address various problem states to improve performance.

In the following sections, we discuss each of the three domains mentioned above. Specifically we discuss their similarities, while contrasting their differences. For each domain, we address how we foresee the role of next generation analysts and technologies in augmenting cognitive and collaborative work, and in turn how that impacts our ability to observe, study, and support said analysts.

2.1 Battle Management / C3I Activities

Our first problem set at the USAF involved NORAD operations and improving analyst performance during the late 1980s. This research provided the initial seeds for studying how individual and teams of analysts use large group displays to make time-constrained decisions involving C3I operations [11]. Battle management inclusive of analyst work (e.g., command, control, communications, and intelligent analyst integration) can lead to many problems within problems, resulting in fatal decisions or strategic triumphs. Battle management also spans across individuals, small teams, and multi-team systems - magnified many times over. Therein, it inherently requires collaborative, cooperative, and coordination activities that produce interdependent information sharing, help solve problems, and enable effective decision making. In this regard, analyst work is dependent on a team’s understanding of their own cognition process, and the process by which they jointly use information analytic functions to plan, allocate, distribute and re-plan their activities. It requires high demands of attention to integrate individual results with other team member outputs. In addition to intense cognitive processing of data and extrapolation to information requirements, an analyst must partake in meta-cognitive processes at both the individual and team levels. Error monitoring, re-planning, and compensational activities when a team member is overloaded are typical kinds of meta-cognitive activities that can differentiate high from low performing teams. Much of their work couples temporal aspects of cognition and planning in being ready to assess situation awareness and make decisions as situations emerge into the future, especially if limited resources cause readjustment of plans. In turn, this area has been informed quite a bit from the Naturalistic Decision Making (NDM) perspective. This context also is responsible for the introduction of many new human-computer interaction advances, computer-supported cooperative work technology, and advanced agent-based decision aids. Hence, it has been a major driver of technology development to provide support for workers and enhance their performance on wicked problems.

2.2 Intelligence / Image Analyst Activities

When it comes to intelligence and image analyst activities, there is substantial emphasis placed on the individual [8, 9]. Because of the nature of this activity, analysts may be more firewalled from each other than they need to be. Because of potential for individual silos, this kind of complex analyst work has historically dampened potential solutions involving direct teamwork. Individual analysts working with intelligence data draw on many sources to ascertain situation awareness and engage in cognitive processes such as perception, reasoning, logic, and argument to frame a given situation and what is impacting it from an intelligence perspective. Their problem sets involve piecing together individual events that comprise a common operational picture and the cause-effect sequences that produce outcomes. An intelligence analyst will be involved in reading, assimilating, interpreting, and producing reports that yield an increase in intelligence and power within the activities under investigation. This kind of activity demands reading comprehension, information seeking, and engaging in sensemaking to distill what is happening in any given situation, the contextual surround, or related situations. Similarly, image analysts engage in information seeking and sensemaking, but rather specializes more on perceptual differentiation and are experts at seeing deep within an image to detect and classify images as they relate to a bigger picture. At times both of these types of activities can be important within battle management (e.g., an all-source analyst may engage in integrative, fusion-based activities to decipher what is going on and to formulate a richer common operational picture). Because misinformation and false alarms are often resident within the cognitive processing cycle, the nature of this kind of work can be difficult. Hence, resiliency and being able to put data together in different ways (repositioning – reperception) can help adroitness. Even though the context of the individual analyst is inherently not directly involved in teamwork, the work does require information sharing (usually asynchronously) with others who have a need to know the information. Collaborative information sharing helps to provide a more realistic, complete picture and can fill-in-the-blanks of hidden scenarios, help to verify sources, and provide greater awareness of a situation developing.

The clandestine nature of the job places a premium on secrecy; collection of information can occur from many different sources around the world in many different languages; and assimilating the collected information into something that is clear and useable can have challenges of its own.

Proc. of SPIE Vol. 8758 87580F-3

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

The unique problem space that Intelligence production presents is discussed by Hastie [12] in depth. Hastie outlines specific issues and challenges that serve to make intelligence a unique realm:

1. “Analysts are removed from many sources of their intelligence

2. Denial and deception must be considered when evaluating credibility and validity of information

3. Many outcomes involve low-probability, high-impact consequences

4. Temptation for bias

5. Sometime pursuing conflicting and non-aligned objectives

6. Feedback is rare and unreliable”

The unique challenges presented by Intelligence production as it relates to information collection and analysis for national security issues also emerge in what to assesses as high priority, highest security. Lowenthal [13] discusses the issue of ‘ad hocs’ which often create “conflicting and competing priorities” for analysts, as new situations of higher priority conflict with ongoing projects.

The varied nature of collaborative activities within Intelligence Analysis makes it a rich platform from which to examine issues of team interactions and team cognition.

2.3 Cyber-Operations Activities

Another such domain of interest in which complex cognitive and collaborative work is often at play is cyber operations. Cyber operations is an emerging domain of interest in multiple disciplines such as Human Factors, Cognitive Modeling, Algorithms, and Visualizations, to name a few. Unlike the relationship between the previous two domains, the cyber security context presents a new and unique challenge to both researchers and practitioners alike. Unlike traditional operating environments, in cyber operations there is disconnect between the analysts, and their cognitive representation of the real-world. Cyber threats are often invisible, providing little to no indication of what is occurring or who is responsible. This type of threat, in turn, creates an inherent disconnect between the analyst and the environment. The native state of a cyber-environment is represented as text-based data that is made up of time-stamps, IP addresses, host names, port numbers, and even codes, making it difficult for analysts to form a holistic picture of cyber events. These cognitive issues become further magnified when scaling to the team or organizational level. It is often the case in cyber operations centers that the awareness of the state of the environment is divided amongst multiple functional domains [10]. Because of this division within the organizational structure, rather than relying on interdependent work, in which analysts closely collaborate with others, they often work independently, only moving to interdependency when there is an overlap in goals. This amounted to analysts passing incident reports, or event analyses, across domains based on who was best suited, or responsible for the type of event. This creates a cognitively intensive secondary task, where analysts must maintain an interdependent awareness of the state of other actors in the system.

While research and work in the previous domains has provided us with invaluable knowledge on how individuals, teams, and technology operate within complex, decision-making environments, the paradigmatic shift is so substantial that the same principals may not apply. When considering the description of warfare above, several unique attributes stand out that separates the cyber battlefield from predecessors. McNeese [6] defines warfare as a struggle between opposing forces, with clear environmental constraints, and limited supplies of resources. The cyber battlefield transcends these notions. These battles are not fought between forces; rather they are often fought by distributed, anonymous2 civilians, who possess the knowledge and resources to execute complex and costly multi-stage cyber-attacks. Likewise, the battlefield of cyber space does not adhere to the same rules and physics as a traditional environment does. The notion of space, time and finite resources do not apply. With this inherently complex, and ever-changing landscape, cyber operations afford an equally as dynamic perspective.

2 The use of “anonymous” is referring to the general anonymity that individuals can achieve while on the internet and is making no reference, claim, or statement about the cyber-activist group of the same name.

Proc. of SPIE Vol. 8758 87580F-4

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Knowledge Elicitation

i

Scaled World Simulations

models

Problem -based Approach

use

Ethnographic Study Reconfigurable Prototypes

3. RESEARCH METHODOLOGY In response to individualistic research methods that tend to produce part-task or piecemeal results, the Living Laboratory (LL) Methodology [14] was implemented to afford integrative understanding and workable solutions that can be triangulated in unique ways to gain power. The LL provides multiple rationales for discovery, design, and insight that simultaneously highlights both science and engineering and praxis in ways that are multiply reflective and expedient. The goal of this philosophy is to be true to ecological diversity in discovering how phenomena is impacted through human intentions, work environment, and the adaptive design of tools that support action in a context to achieve intentions. The Living Lab draws upon ideas laid down by proponents of cognitive and ecological systems [15]. It develops through the inculcation of multiple methodologies applied and integrated as needed to understand context and phenomena in the world. The LL is just one instantiation of a larger set of frameworks that draw from a similar genre of approaches to study humans in their environments (human-agent mutuality) [16]. Figure 1 provides a graphical representation of the Living Lab.

Figure 1: Visual Representation of the Living Lab Framework

More specifically, the Living Lab Framework utilizes observations of humans within their work domain alongside a deeper understanding of their knowledge to aid in the design and implementation of realistic simulations to be used as test theory and technology in a controlled setting. Results from laboratory studies can then be taken back out into the field and implemented into the actual environment. During this process, researchers can use their findings to help further theory, model and tool development. While presented visually and described as a cyclical process, the Living Lab Framework is flexible enough that one can move between the components forwards, backwards, crosswise, or even perform them in conjunction with each other.

The living lab, itself is particularly suited for next-generation analyst research due to its ability to leverage multiple methods in triangulating the entirety of the problem space within a given context or environment. Through the living lab, one can view analyst work through multiple lenses. We can gain an understanding of the greater environment, and the varying confounds at play through qualitative observations, while honing in on the cognitive work through in lab experimentation and scaled worlds. This perspective can not only improve the design of technology, but can also be useful in informing the development of future training programs, team and individual interventions, and performance metrics, all which can be useful in perpetuating our national goals.

3.1 Foundation Elements: Theory-Problems-Practice

Much of the LL approach begins with the practice. We briefly described some of the nature of work from the three analysts settings where we gathered real world information as a basis for formulating this paper. Practice is a beginning point for understanding analyst’s work in terms of what it is, but also as a basis for what it can become if change is resident in work. In this light, practice is the beginning step to a bottom-up approach to understanding intention-agency- action-relevance. At the heart of the LL is the idea of problem-centered learning. Problems often emerge from practice and represent issues or constraints on actions that workers deal with. When people overcome their problems there is a great amount of learning that can be reused or apportioned for other aspects of their work. Therein, problems facilitate

Proc. of SPIE Vol. 8758 87580F-5

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

learning in new ways to reveal thinking that may be promising. Problems can be difficult, hard to overcome, and almost impossible but their place in the context of work provides the basis for exploring new ways of change that help adapt technology to support human intentions. Problems carry with them reflections towards all aspects of the LL and that is why they are graphically at the center of the framework. As practice elicits problems that help guide a researcher to understand where research might aim, theory provides a top-down process that also intersects problems. Theoretical stances may explain what caused a problem or why a problem persists, provide a larger perspective on a problem, predict when a problem is salient, and generate scientific literature that documents current research on a problem. Hence, problems can be informed from findings in the theoretical and practical areas as feedback for knowing where to aim future research.

3.2 Practical Elements: Ethnography-Elicitation-Simulation-Reconfigurable Design

Complementing the previous vertical elements of the LL are the horizontal elements that move from left to right. These practical elements provide the means to further explore theory-problems-practice and represent process methods that provide additional knowledge to bear upon human-systems integration. The cohesiveness of these methods is important in that the goal is to establish reflective triangulation and learn from diverse angles how problems can be overcome in efficient, resilient, and effective ways that capture multiple perspectives on solutions. Many of these methods yield feedback to strengthen each other and reinforce concepts, ideas, and innovation. Some of the original ideas concerning integrative representation structures for complex work came from the AKADAM framework [17], wherein knowledge contained as mental models were elicited from experts using concept mapping techniques [18]. Concept maps were integrated with functional work descriptions (IDEF models) and design storyboards (a prototyping tool) to produce “knowledge as design”. These representations then can be the basis for experimental research and the design of interfaces. We believe in the integration of information using several tools to produce more informational power, rather than a single tool alone. In turn, the AKADAM suite was still too limited. Building off of this, we expanded AKADAM into the Living Laboratory to provide broader bandwidth coverage and to incorporate some tools. As mentioned above, outputs from one element can be the basis for input into another (feed-forward) and therein powerfully inform design from collective induction providing an in-depth understanding from a broad bandwidth.

The next section will provide examples of how these elements have been used to gain comprehension of analyst’s work from the different domains we described earlier. Showing select examples of product output from each domain helps to show the reader what the LL provides for basic and applied research environments.

4. RESULTS: APPLICATIONS AND PRODUCTS Looking at complex analyst work has provide many opportunities to both understand and improve support of analyst function whether it be for BMC3I (Battle Management, Command, Control, Communications and Intelligence), Intelligence/Image Analyst Work, or Cyber-Security Operations. In the previous sections we have discussed how our approach focuses on learning from practice and talking to experts. In addition to this, the other cornerstone of the framework is in building scaled world simulations to represent many of the other elements in one comprehensive orchestration. Building these kind of simulations is non-trivial, but allows much exploration, testing, and validation of theories, ideas, information, and technologies that come from other elements of the LL. Therein, many of the products, applications, and information flow draw back into the scaled worlds for reuse and for further capability upgrades. Table 1 below provides an array of results for our joint efforts to date in developing scaled world simulations.

Proc. of SPIE Vol. 8758 87580F-6

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

Version Description Theories Technologies

1.0 First version of NeoCITIES. Informed by original CITIES simulations and ethnographic study of emergency dispatch centers

• Team Mental Models

• Situation Awareness

• Stress and Mood

• Intelligent Group Interfaces

• Fuzzy Cognitive Maps

2.0 Updated version of 1.0 with new technologies. Designed as a part of collaboration with Geo- Visualization researchers.

• Team collaboration • Situation

Awareness

• Geo-Collaborative interface

• Shared workspace

3.x Streamlined previous simulations for experimental research. Greater emphasis on mechanisims to study varying aspects of team cognition

• Information Overload

• Situation Awareness

• Team Mental Models

• Collaborative Interfaces • Virtual Storytelling

4.0 (NETS)

Redesign of NeoCITIES 3.x to support the study of cyber security analysts and teams. Informed by interviews and observations of cyber security analysts and exercises

• Situation Awareness

• Workload • Task prioritization • Transactive

Memory

• Cyber visualization • Workload previews • Transactive Memory

Interfaces

Table 1. Results of Living Laboratory for Complex Analyst Work

As the previous table demonstrates, NeoCITIES simulations have been an effective test-bed for research at the Pennsylvania State University, College of Information Sciences and Technology for well over a decade. The initial NeoCITIES simulation was developed based on an ethnographic study and knowledge elicitation based on 911 dispatchers. Using the knowledge gained from this research, the first iteration, NeoCITIES 1.0 was designed and deployed in several different experimental research projects. This version served as a platform for studying team decision in emergency response, as well as informing the design of interfaces to support these interactions. Following research on NeoCITIES 1.0, we collaborated with researchers focused on Geographic Information Systems to develop a follow up simulation, NeoCITIES 2.0. This simulation focused much more on the technology, and was developed primarily as a demo for geo-visualizations and as a task to study geographic collaborations. The subsequent simulation, NeoCITIES 3.0, continued to utilize similar technological platforms as 2.0, but refocused the emphasis away from visualizations, and rather focused on extracting various aspects of team cognition for basic research. Finally, in the past few years, attention within the MINDS group has shifted away from emergency response, and towards the dynamic and emergent domain of cyber security. In response to this, the newest iteration NeoCITIES 4.0 (a.k.a. NeoCITIES Experimental Task Simulation; NETS) was developed based on an ethnographic study of cyber security analysts. To account for the breadth of the domain, NETS was developed with flexibility and extensibility in mind. To date, NETS has been in multiple studies focusing on individual and team cognition, as well as the development of visualizations and interface artifacts.

5. LOOKING TOWARDS THE FUTURE Referencing another historical review our research group did back in 1998 [6], we looked at the historical precedence and application significance of teamwork, team performance, and team interfaces within the USAF. In that review we traced the zeitgeist that informed research for over a period of forty years, and in turn examined underlying team research and design/development trajectories that ranged from - psychology in the 1960s - to technology in the 1970s – to fieldwork in the 1980s – to cognitive systems engineering in the 1990s – to socio-culture in the 2000s. We have now surpassed 2010 and need to revise the prognostication with respect to this decade, especially as it relates to next

Proc. of SPIE Vol. 8758 87580F-7

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

generation analyst work. The following sections outline new perspectives and visions that we believe will be salient for future work.

5.1 Cultural Aspects of Analyst Work

The role of intelligence is to reduce uncertainty for decision-makers; however, in the post-9/11 era, standards for intelligence have become “unreasonable”3. By implication, there is increasing scrutiny of the analysts’ necessary skills. Moore [19] identifies knowledge, abilities, characteristics, and skills as the “core competencies” for analysts. In other words, analysts’ competencies are a combination of personality skills (i.e. individual characteristics) that cannot be taught and three areas (i.e. knowledge, abilities, and skills) that may be developed through formal education and experience. Knowledge competencies reflect “learned knowledge” [19] that contributes to historical or broad contextual knowledge and more operational or tactical situational awareness of a “target environment.” The knowledge and personality competencies combine with abilities (i.e., communication and collaboration) and skills (i.e., critical thinking, computer literacy, project management, and language) to create the competent analyst. Moore contends that while all competencies would be ideal, analyst may produce valuable intelligence without selected competencies dependent upon the kind of intelligence they are being asked to produce [19].

Although not specifically addressed in Moore’s work, there is an emerging debate to broaden the humanistic agenda focused on “critical/cultural” perspectives for intelligence studies [20]. From a cultural perspective, more competencies rather than fewer are required. Cultural knowledge can make “descriptive” and “explanatory” intelligence reports more robust but is critical to fulfilling higher level interpretative and estimative” intelligence requirements. This cultural knowledge is a requirement to explain the importance of information, to predict future states, and to recognize patterns and trends. While this does not necessarily need to be inculcated in every individual analyst, cultural expertise, including linguistic proficiency, is a prerequisite for collaborative teams working on interpretative and estimative intelligence products.

True to predictions, the post 9/11 era has increased the demand for multinational analyst teams in support of coalition warfighting, cyber-security, and intelligence operations in support of counter-terrorism. Geographical disbursed teams working on the same problem have further complicated the integration of teams and the sharing of information. This created an explicit need for research to understand the effects of cultural differences in decision-making, problem solving, and in other human interactions and behaviors. Cultural diversity is essential to improving situational awareness but also presents challenges that can inhibit multicultural teams from reaching their common goal. The embedded nature of culture on human behavior and cognition make it difficult to pinpoint it effects, but it can have a dramatic effect on outlooks, approaches, perspectives, attitudes and values of individuals without them being directly aware of the unknown biases and perceptions. This embedded nature of culture can then complicate face-to-face interactions, impacting interpersonal communication and the dynamics of collaborative activities. In distributed, asynchronous situations the impact of cultural differences can have profound effects on the outcomes of group activities, without team members being aware of it until there are consequences.

A need, then, arises to understand the ways in which culture emerges on an individual and team level and the ways in which it can affect outcomes and performance, particularly on group decision making and problem solving behaviors. Cultural diversity among team members implicates a different nuance to team behaviors and interactions, not all of these are negative, however. For example, a review of the literature reveals several characteristics of heterogeneous cultured teams such as creativity in problem solving [21], an ability to solve complex problems with differing views, avoid group think, and outperform homogenous cultured teams. The diversity inherent in heterogeneous cultured teams creates a diversity of perspectives in problem solving tactics. Complex problems often require creative solutions, which in turn is facilitated via diversity, cultural or otherwise. Compared to homogeneous groups, heterogeneous groups tend to make more creative and higher quality decisions [21]. It is this feature that makes cultural diversity an asset to team processes. The benefit of diversity is that it also leads members to challenge one another and to incorporate a variety of perspectives into the problems solving efforts of the group– allowing them to often come up with the better solution than homogenous teams.

Complicating factors introduced by culture include that differences in work norms and behaviors create strife, communication barriers (such as language differences) affect group performances, which can lead to a higher potential for conflict, and more stress for team members. These negative effects can be even more profound with the added

3 Remarks given by James Clapper to the US House Permanent Select Committee on Intelligence, February 10, 2011

Proc. of SPIE Vol. 8758 87580F-8

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

complexity that emerges in distributed or asynchronous analyst teams with a multicultural make up and can create a large number of challenges that will greatly affect the outcomes of team performance.

5.2 Collaborative Information Seeking

One area where technology will enable new approaches to influence teamwork, culture and new theoretical precedence in is the area of collaborative information seeking (CIS). CIS is a specific process within the overarching team decision- making process, aimed at a set of activities taken by a team or organization to identify and resolve a shared information need [22]. Often, within team decision-making models, the process of CIS is overlooked and incorporated within the broader notion of decision-making. Within the context of next generation analysts, CIS needs to be a focus. The deluge of information brought forth from the influx of teamwork and advances in technology makes CIS a critical activity of next generation analysts. Seeking, collaborating, and sharing information is more paramount now than ever for analysts. CIS will only become a more pressing and important process as next generation analysts are given the responsibilities of working within more teams and subsequently finding and understanding more information within and among teams.

Traditionally (pre 9/11), analysts worked within individual silos, sharing little information. After 9/11 the paradigm changed, leading to an influx in teamwork and sharing of information across boundaries. While this enhanced collaboration within teams and across organizations is viewed as a positive, it brings forth the knowing and ability to accurately seek and share information. Thus, next generation analysts must be proficient and experts in CIS. As analysts seek and share information with each other, they take part in two distinct cognitive processes, information comprehension and knowledge creation (e.g., collective induction). These complementary yet competing processes create unique challenges in both research and practice. A team has the ability to exponentially increase the amount of information found, but understanding the information that each team member has brought forth from a high-level team perspective is challenging. Next generation analysts must not only know how to search information individually, but search for information in a collaborative manner and then synthesize all information found from all team members to understand the holistic contextual and informational landscape.

In addition to the growth of teams within analyst’s work, the influx of new technologies, specifically socio-cognitive, has led to CIS being a central activity. Multiple technologies have been produced within recent years that help teams and organizations seek and share information, allowing for advances within CIS. Yet, technology often brings forth more information, meaning that a team must work to understand that information. While information enabled technologies are no doubt helping analysts find information, they are also creating difficulties related to information overload. A human or group of humans can only cognitively process so much information at once. Often, technologies result in information overload, consequently hurting the CIS process. This paradigm means that next generation analysts must understand and utilize technology to collaboratively find information. But, at the same time be able to discern what information is critical to the task at hand without being cognitively overloaded by the sheer amount of information being brought forth from the team and the technology.

The abundance of information brought forth by teams and technologies are paving the way for the why CIS is and will become an integral activity for next generation analysts, but additional issues and characteristics must also be considered. The changing contextual and informational landscape among analysts brings forth new perspectives on collaborative information seeking and growing importance.

5.3 Evolving with Socio-Cognitive Advances

During the initial conception of the Living Lab Framework the landscape of command centers were substantially different from current configurations. At that time, command centers would rely on analog connections with each other, which could relay a handful of channels of audio, and potentially video. In this iteration of the command center, analysts metaphorically existed in a box, where the majority of their connections remained inside the room, with minimal contact outside. As technology evolved, so did the command center. Today, if you were to walk into a modern command center, you could find a cornucopia of new technologies aimed at not only aiding in individual decision support, but also connecting analysts across geographic, temporal and cultural boundaries to aid in planning and decision-making. This influx of technology has empowered analysts by supporting and encouraging such cognitive and collaborative behaviors such as distributed cognition, transactive memory formation, and team situation awareness, to name a few

As discussed in the previous two sections, evolutions in the technological, as well as political landscape have brought forth number of changes that may have a dramatic effect on the team and task work processes that occur in everyday analyst work. As previously mentioned, multi-national teams are growing within analysts work. As these teams are often

Proc. of SPIE Vol. 8758 87580F-9

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

separated by geographic and temporal boundaries, the nature of their work, and medium by which they achieve it is constantly in flux. A multi-national team inherently creates very different mindsets in regard to searching information, which produces incongruence and inefficiencies during the CIS process. This problem is confounded as more and more new technology enters the command center. Additionally, members within a multi-national team often interpret and understand found information in very different ways. Thus, next generation analysts must understand these differences at the onset of the teamwork and know how to overcome them, or the team will not find the requisite information. In addition to multi-national teams, problems often arise from the incompatibilities between older and newer analysts. Both have set ways of finding, sharing, and understanding information. Older analysts (not all) rely more on individual work and traditional means of finding information, whereas newer analysts prefer work within teams and find most information by utilizing technology. Next generation analysts must understand these information related differences, and know how to find and share information with old and new analysts, almost becoming the connection between old and new preferences for work.

While this incorporation of technology and multi-national has made information available to analysts like never before, it is a double-edged sword. As more technology, and thus information, enters into the command center, analysts are forced to deal with new cognitive activities, overcoming the deluge of information created by the technology, and collaborations. Since this evolution, which is for all intents and purposes, uncharted territory, it is difficult to know the best way to design and incorporate such technologies, and to understand the confounding affects they may have on the analysts. In this perspective, the Living Lab is especially suited to serve as a methodology to address these issues. Using the living lab, designers and researchers can hone in on technological, collaborative, and cognitive behaviors that are at play, and blend their findings across multiple phases of research. Though our own work with the Living Lab Framework, we have noticed the evolution, and degrees of differences, on how the Living Lab Framework has been put into practice. Clearly, this is the case for most environments, where their individual differences generate constraints on the overall framework. Therein, the adaptivity of the framework has becomes it’s greatest asset, which is continually reinforced overtime. Within each of these next-generation analysts contexts, due to the influx of new technologies, multi-cultural teams, and new collaborative information seeking processes, several cognitive, social, and political issues will arise, which in turn will have an effect on the implementation of the Living Lab for that specific case. In turn, the central node, of problem (and sub-problem) identification and definitions becomes a strategic element in establishing ground truth of the specific approach being taken for any given domain of interest. Taking the time to explore problems given the nature of the human and the changing contextual, cultural and technological surroundings acts to ground the approach to the problem space.

6. CONCLUDING REMARKS At the heart of this paper is the idea that technology influences the human, and that humans influence each other as well as technology, which in turn impacts performance efficiency and effectiveness. As technology continues to evolve, the impact of technology on humans will continue to become more and more significant. Humans in environments such as C3I, emergency response, and cyber security will continue to lean on technology more and more. As humans rely more and more on technology, distributed cognitive processes will emerge causing their everyday work to become increasingly invisible. This paradigmatic shift causes major concerns for both ethnographic and experimental researchers alike. With invisible work comes an inherent difficulty in assessing what is actually occurring. From the naked eye it may seem like complex cognitive work may have decreased as a result of the technology, but in reality it has just shifted. As discussed earlier, while technology such as visualizations, collaborative systems, and social media, may offload traditional work from the analyst, it creates new requirements of the job, which are not directly observable and are often not intuitive. For example, a collaborative system with shared workspaces may remove the physical task of “hand-offs,” but it also creates a new task of maintaining an interpersonal awareness of the status of other work within the system. Likewise, while social media may remove the task of information seeking through multiple sources, it also creates a deluge of information, which in turn takes up cognitive space, which may typically be allocated for other functions. In no way are we suggesting that these technologies are in any way detrimental, but rather we are proposing that they are changing the course of work, creating new, dynamic and emergent cognitive behaviors that we must account for.

Proc. of SPIE Vol. 8758 87580F-10

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

This claim, which we have made over the course of this paper, is not a new postulation. In fact, this idea is a pillar of research in fields such as Human Factors, Human-Computer Interaction, and Computer-Supported Cooperative Work. However, while many researchers are quick to accept this evolution, their methodologies and approaches rarely reflect the changing environments. While traditional methodologies of studying cognition, and assessing the utility of technology, may provide some insight, they are often incapable of painting a full and holistic representation of the environment, and the actual cognitive work that occurs within it. It is with this that we have proposed the Living Lab Framework as an outline for such research moving forward. The true power of the Living Lab is its flexibility. What we have presented here is merely an outline, which should, and must be adapted, modified, and tweaked to account for individual differences and nature of the research, environment, and individuals within.

REFERENCES [1] Sarter, N. B., Woods, D. D., & Billings, C. (1997). Automation surprises. In G.Salvendy (Ed.), Handbook of human

factors/ergonomics (2nd ed., pp. 1926–1943). [2] Woods, D. D., Johannesen, L. J., Cook, R. I., & Sarter, N. B. (1994). Behind human error: Cognitive systems,

computers, and hindsight. Wright Patterson AFB, OH: CSERIAC. [3] Rasmussen, J., Pejtersen, A. M., & Goodstein, L. P. (1994). Cognitive systems engineering. New York: J. Wiley. [4] Klein, G. (1993). A Recognition-primed decision (RPD) model of rapid decision making. In G. Klein, J. Orasanu,

R. Calderwood & C. E. Zsambok (Eds.), Decision making in action: Models and methods (pp. 138-147). Norwood, NJ: Ablex Publishing Corporation.

[5] McNeese, M. D. (1996). Collaborative systems research: Establishing ecological approaches through the living laboratory. Proceedings of the 40th Annual Meeting of the Human Factors Society,2, (pp. 767-771). Santa Monica, CA: Human Factors Society.

[6] McNeese, M. D. (1998). Teamwork, team performance, & team interfaces: Historical precedence and application significance of the research at the USAF Fitts Human Engineering Division. Proceedings of the IEEE International Symposium on Technology and Society (pp. 161-166), IEEE Society on Social Implications of Technology, South Bend, IN.

[7] McNeese, M. D., Rentsch, J. R., & Perusich, K. (2000). Modeling, measuring, and mediating teamwork: The use of fuzzy cognitive maps and team member schema similarity to enhance BMC3I decision making. IEEE International Conference on Systems, Man, and Cybernetics, (pp. 1081-1086). New York: Institute of Electrical and Electronic Engineers.

[8] McNeese, M. D., Connors, E., Jefferson, T., Bains, P., Jones, R., Terrell I., Brewer I., Craven, P., Van Vactor, J., Rotthoff, E., & Hall, D. (2004). An assessment of image analyst work using the living laboratory framework (Tech. Rep. No. 0017). University Park, Pennsylvania: Pennsylvania State University, School of Information Sciences and Technologies. Report to the National Geospatial-intelligence Agency (NGA). Washington DC.

[9] Connors, E. S., Craven, P. L., McNeese, M. D., Jefferson, Jr., T., Bains, P., & Hall, D. L. (2004). An application of the AKADAM approach to intelligence analyst work. Proceedings of the 48th Annual Meeting of the Human Factors and Ergonomics Society (pp. 627-630). Santa Monica CA: Human Factors and Ergonomics Society.

[10] Tyworth, M., Giacobe, N., Mancuso, V., & Dancy, C. (2012). The distributed nature of cyber situation awareness. 2012 IEEE Conference on Cognitive Methods in Situation Awareness and Decision Support.

[11] McNeese, M. D., & Brown, C. E. (1986). Large group displays and team performance: An evaluation and projection of guidelines, research, and technologies. AAMRL-TR-86-035. Armstrong Aerospace Medical Research Laboratory, Wright-Patterson Air Force Base, OH.

[12] Hastie, R. (2011). Group processes in intelligence analysis. In B. Fischhoff & C. Chauvin (Eds.), Intelligence Analysis: Behavioral and Social Scientific Foundations (pp. 169-196). Washington, D.C.: The National Academies Press.

[13] Lowenthal, M. M. (2008). Intelligence: From secrets to policy (4th ed.): CQ Press [14] McNeese, M. D., Pfaff, M. S., & Mohammed, S. (2011). Looking at macrocognition through an interdisciplinary,

emergent research nexus. In E. Salas & S. Fiore (Eds.), Theories of team cognition: Cross-disciplinary perspectives. London: Taylor & Francis.

[15] Vicente, K. J. & Rasmussen, J. (1990). The ecology of human-machine systems II: Mediating "direct perception" in complex work domains. Ecological Psychology, 2, 207-250.

Proc. of SPIE Vol. 8758 87580F-11

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

[16] Niitamo, V-P., Kulkki, S., Eriksson, M., & Hribernik, K.A. (2006). State-of-the-art and good practice in the field of living labs. The 12th International Conference on Concurrent Enterprising: Innovative Products and Services through Collaborative Networks, ICE 2006, Milan, Italy, June 26-28, (pp. 349-357).

[18] Zaff , B. S., McNeese, M. D., & Snyder, D. E. (1993). Capturing multiple perspectives: A user-centered approach to knowledge acquisition. Knowledge Acquisition, 5, 79–116.

[19] Moore, D. T. (2002). Species of competencies for intelligence analysis. Defense Intelligence Journal 11(2), 8. [20] Bean, H. (2012). Rhetorical and critical/cultural intelligence studies. Intelligence and National Security, 09 July

2012, p.3. [21] van Vliet, A. J., & van Amelsfoort, D. (2008). Multinational military teams. Multinational Military Operations and

Intercultural Factors NATO OTAN. [22] Poltrock, S., Grudin, J., Dumais, S., Fidel, R., Bruce, H., & Pejtersen, A. M. (2003). Information seeking and

sharing in design teams. In Proceedings of the 2003 international ACM SIGGROUP conference on supporting group work (pp. 239-47).

Acknowledgements Work on this paper was partially supported by U.S. Army Research Office (ARO) MURI Grant “Computer Aided Human Centric Cyber Situation Awareness” W911-NF-09-1-0525. The Opinions expressed in this paper are those of the authors only and do not necessary represent the official position of the Army Research Office, the US Army, the Department of Defense or the Pennsylvania State University. We also gratefully acknowledge that this research activity has been supported in part by a Multidisciplinary University Research Initiative (MURI) grant (Number W911NF-09-1-0392) for “Unified Research on Network-based Hard/Soft Information Fusion”, issued by the US Army Research Office (ARO) under the program management of Dr. John Lavery.

Proc. of SPIE Vol. 8758 87580F-12

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/21/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx