Integration of JIIM Roles
Tie3D
JIIM.pdf
U.S. Army Research Institute
For the Behavioral and Social Sciences
Research Report 1963
Training Joint, Interagency, Intergovernmental, and Multinational (JIIM) Participants for Stability Operations
James C. Ong Stottler Henke Associates, Inc.
Karol G. Ross
Cognitive Performance Group
Brooke Schaab U.S. Army Research Institute
Mike Prevou and Holly Baxter Strategic Knowledge Systems
Anna Grome
Klein Associates Division, ARA
David Spangler Global Innovation and Design, Inc.
Julia Loughran ThoughtLink, Inc.
September 2012
Approved for public release; distribution is unlimited
U.S. Army Research Institute for the Behavioral and Social Sciences Department of the Army Deputy Chief of Staff, G1 Authorized and approved for distribution:
BARBARA A. BLACK, Ph.D. MICHELLE SAMS, Ph.D. Research Program Manager Director Training and Leader Development Division Research accomplished under contract for the U.S. Army Research Institute Stottler Henke Associates, Inc. Technical review by COL Robert C. Morris, Jr. J. Douglas Dressel
NOTICES
DISTRIBUTION: Primary distribution of this Research Report has been made by ARI. Please address correspondence concerning distribution of reports to: U.S. Army Research Institute for the Behavioral and Social Sciences, ATTN: DAPE-ARI-ZXM, 6000 6th Street (Bldg. 1464 / Mail Stop 5610), Ft. Belvoir, VA 22060-5610. FINAL DISPOSITION: Destroy this Research Report when it is no longer needed. Do not return it to the U.S. Army Research Institute for the Behavioral and Social Sciences. NOTE: The findings in this report are not to be construed as an official Department of the Army position, unless so designated by other authorized documents.
i
REPORT DOCUMENTATION PAGE 1. REPORT DATE (dd-mm-yy)
September 2012
2. REPORT TYPE Final
3. DATES COVERED (from - to) September 2007 - May 2010
4. TITLE AND SUBTITLE Training Joint, Interagency, Intergovernmental, and Multinational (JIIM) Participants for Stability Operations
5a. CONTRACT OR GRANT NUMBER 5b. PROGRAM ELEMENT NUMBER
622785 6. AUTHOR(S): James C. Ong, Karol G. Ross, Brooke Schaab, Mike Prevou, Holly Baxter, Anna Grome, David Spangler, and Julia Loughran
5c. PROJECT NUMBER A790
5d. TASK NUMBER 5e. WORK UNIT NUMBER
366 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Stottler Henke Associates, Inc. 951 Mariner’s Island Blvd., Suite 360, San Mateo, CA. 94404
8. PERFORMING ORGANIZATION REPORT NUMBER
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) U. S. Army Research Institute for the Behavioral & Social Sciences 6000 6TH Street (Bldg. 1464 / Mail Stop 5610) Fort Belvoir, VA 22060-5610
10. MONITOR ACRONYM ARI
11. MONITOR REPORT NUMBER Research Report 1963
12. DISTRIBUTION/AVAILABILITY STATEMENT Distribution Statement A: Approved for public release; distribution is unlimited. 13. SUPPLEMENTARY NOTES Contracting Officer’s Representative and Subject Matter POC: Angela Karrasch 14. ABSTRACT (Maximum 200 words): This project supports training Joint, Interagency, Intergovernmental and Multinational (JIIM) participants for planning and implementing Stability Operations. Based on a literature review and subject matter expert interviews, a set of themes was developed that reflect the high-level cognitive skills that experts use to be successful in JIIM environments. Then a collection of computer-based training tutorials was developed and then pilot-tested by students at the U.S. Army Command and General Staff College (CGSC). Traditional training systems present information and then test their recall. By contrast, these tutorials interweave the learning of FM 3-07 and other doctrine with problems and scenarios. This approach has been shown to accelerate learning by providing scenario-based learning goals and contexts for acquiring, integrating, and retaining declarative knowledge more effectively.
15. SUBJECT TERMS Computer-based training, stability operations, accelerated training, scenario, Socratic dialog, Joint Interagency Intergovernmental Multinational (JIIM) SECURITY CLASSIFICATION OF 19.
LIMITATION OF ABSTRACT
Unlimited
20. NUMBER OF PAGES
54
21. RESPONSIBLE PERSON
Dorothy E. Young (703) 545-2316
16. REPORT Unclassified
17. ABSTRACT Unclassified
18. THIS PAGE Unclassified
ii
iii
Research Report 1963
Training Joint, Interagency, Intergovernmental, and Multinational (JIIM) Participants for Stability Operations
James C. Ong Stottler Henke Associates, Inc.
Karol G. Ross
Cognitive Performance Group
Brooke Schaab U.S. Army Research Institute
Mike Prevou and Holly Baxter Strategic Knowledge Systems
Anna Grome
Klein Associates Division, ARA
David Spangler Global Innovation and Design, Inc.
Julia Loughran ThoughtLink, Inc.
Fort Leavenworth Research Unit
James W. Lussier, Chief
U.S. Army Research Institute for the Behavioral and Social Sciences 6000 6th Street, Bldg. 1464, Fort Belvoir, VA 22060
September 2012
Army Project Number Personnel, Performance 622785A790 and Training Technology
Approved for public release; distribution is unlimited.
iv
ACKNOWLEDGEMENTS
We are grateful to Prof. Joseph Bebel at the U.S. Army Command and General Staff
College (CGSC) and Prof. Jon Stull at the Joint Forces Staff College who allowed us to pilot-test the tutorials during their courses. We also thank the individuals and organizations such as the U.S. Army 95th Civil Affairs Brigade who helped by participating in subject matter expert interviews or by testing the tutorials and providing their reactions. We are particularly grateful to LTC Hakan Petersson, a Swedish Liaison Officer and member of the Swedish Defence Research Agency, who participated in the Advanced Stability Operations class at CGSC as a subject matter expert. His insightful feedback led to significant improvements in the training.
v
TRAINING JOINT, INTERAGENCY, INTERGOVERNMENTAL, AND MULTINATIONAL (JIIM) PARTICIPANTS FOR STABILITY OPERATIONS EXECUTIVE SUMMARY Research Requirement: Military leaders are taking on new and varied missions. The amount of information required to maintain competence is sometimes overwhelming and cannot be learned during class time in a school house. The requirement for a self-paced, scenario-based training tool that can be updated to maintain relevance on new mission sets provided the stimulus for this research. Procedure:
Guided by the training requirements and design concept, a set of computer-based training tutorials called the Stability Operations in Joint, Interagency, Intergovernmental, and Multinational (JIIM) Environments Tutor was developed. Three versions of the tutorials were developed iteratively. Each version was pilot-tested by students attending the Advanced Stability Operations course at the U.S. Army Command and General Staff College (CGSC). After each version was used by the class, feedback was gathered from students and instructors to identify improvements which were incorporated in the next version of the tutorials. The tutorials include:
• Cross-Cultural Competence Inventory • Introduction to Operations in JIIM Environments • Interagency Management System in Action! • Security in Sudan • Governance and Justice in Haiti • Humanitarian Assistance and Economic Development in Afghanistan
Traditional training systems present information and then test their recall. By contrast, the tutorials interweave the learning of FM 3-07 and other doctrine with problems and scenarios. These scenarios challenge students to apply their knowledge and cognitive skills to assess situations and generate and evaluate options in representative situations. This training approach accelerates learning by providing scenario-based learning goals and contexts for acquiring, integrating, and retaining declarative knowledge more effectively. The scenarios also provide practice opportunities for applying high-level cognitive skills described by the JIIM themes.
To enable instructional developers to create and maintain these types of training systems, an authoring tool was developed that was built on top of an existing technology. Like other authoring tools for computer-based training, this tool enables authors to create a sequence of screens that present information using formatted text and graphics, pose questions, and provide feedback. In addition, the authoring tool simplifies the inclusion of logic that selects and presents information, hints, and feedback adaptively, based upon the student’s previous answers.
vi
Findings:
Training effectiveness was indicated by evaluating the tutorial that taught the Interagency Management System by analyzing pre- and post-test scores and participant responses. Participant responses were positive, however negative responses indicated that “the tutorial took too long to complete.”
In addition to its utility as a training system, the tutorials served as a proof of concept that illustrates the feasibility and effectiveness of this type of computer-based instruction for accelerating introductory learning. It also served as a pilot system that was analyzed to identify ways of improving its instruction. Experience developing the tutorials also guided the development of the authoring tool for creating this type of instruction more easily. The authoring tool was reviewed by a former instructor at Joint Forces Staff College who concluded that the tool could create useful instructional content and could be used effectively by instructors. Utilization and Dissemination of Findings:
The tutorials have been selected for use by the Joint, Interagency, and Multinational Planner's Course (JIMPC), offered at Joint Forces Staff College and the Advanced Stability Operations Course at CGSC. The Office of the Secretary of State is using portions of the tutorial to train civil-military collaboration. The IMS Tutorial has been reviewed by members of the Office of the Secretary of Defense (OSD) Personnel and Readiness. It is under consideration by US JFCOM to educate Combatant Commander and Joint Task Force staffs on the Interagency Management System who cannot attend the S/CRS Foundations course and must rely on distance learning methods. The tutorials are also being evaluated for wider use by other schools at Joint Forces Staff College and by the Swedish Defence Research Agency as a tool for collaboration in Stability Operations.
vii
TRAINING JOINT, INTERAGENCY, INTERGOVERNMENTAL, AND MULTINATIONAL (JIIM) PARTICIPANTS FOR STABILITY OPERATIONS CONTENTS Page
INTRODUCTION .............................................................................................................. 1 METHODS ......................................................................................................................... 2 Domain Analysis ................................................................................................................. 2 Development of Training Themes ...................................................................................... 2 Design Principles ................................................................................................................ 3 Development Process .......................................................................................................... 6 Evaluation and Review ....................................................................................................... 6
Participants .............................................................................................................. 6 Materials ................................................................................................................. 7 Procedure ................................................................................................................ 7
Authoring Tool Development and Usability Assessment ................................................... 8 Tutorials .............................................................................................................................. 8
Tutorial Version 1 – October 2008 ......................................................................... 8 Tutorial Version 2 – May 2009 ............................................................................... 10 Tutorial Version 3 – November 2009 ..................................................................... 11 Tutorial Interactions ................................................................................................ 12 Knowledge Checks ................................................................................................. 17 Situation Judgment Tests ........................................................................................ 17
FINDINGS AND DISCUSSION ........................................................................................ 20 Training Effectiveness Evaluation ...................................................................................... 20
Pre- and Post-test Results........................................................................................ 20 Reaction Survey Results ......................................................................................... 20 Limitations of the Findings ..................................................................................... 23 Authoring tool usability Assessment ...................................................................... 24
CONCLUSIONS AND RECOMMENDATIONS ............................................................. 24 Transition to Operational Use ............................................................................................. 25 Recommendations ............................................................................................................... 25 REFERENCES ................................................................................................................... 27
viii
CONTENTS (continued) Page
APPENDIX A: IMS TUTORIAL KNOWLEDGE TEST ITEMS AND SIGNIFICANCE VALUES FOR DISCRIMINATING PRE- AND POST-TEST SCORES………………. A-1 APPENDIX B: TASK GUIDE AUTHORING TOOL OVERVIEW……………………. B-1 APPENDIX C: AUTHORING TOOL USABILITY QUESTIONNAIRE AND RESPONSES……………………………………………………………………………… C-1
LIST OF FIGURES FIGURE 1. STABILITY TASKS LINKED TO STABILITY SECTORS ........................ 3 FIGURE 2. SECURITY IN SUDAN TUTORIAL PROMPTS FOR USEFUL SKILLS TO INCLUDE IN THE INTEGRATION PLANNING CELL .......................................... 12 FIGURE 3. SECURITY IN SUDAN TUTORIAL POSES A QUESTION TO GUIDE THE STUDENT’S THINKING AND TO MOTIVATE THE ACQUISITION OF COUNTRY-SPECIFIC AND DOCTRINAL KNOWLEDGE .......................................... 13 FIGURE 4. SECURITY IN SUDAN TUTORIAL PROVIDES A REFERENCE (OR EXPERT) ANSWER THAT STUDENTS CAN COMPARE WITH THEIR OWN ANSWER. ........................................................................................................................... 14 FIGURE 5. SECURITY IN SUDAN TUTORIAL RE-PROMPTS TO SELECT USEFUL IPC SKILLS ........................................................................................................ 15 FIGURE 6. SECURITY IN SUDAN TUTORIAL PROVIDES FEEDBACK AND RATIONALE FOR A MULTIPLE CHOICE QUESTION ............................................... 16 FIGURE 7. EXAMPLE SITUATIONS PROMPT STUDENTS TO IDENTIFY RELEVANT THEMES ...................................................................................................... 17 FIGURE 8. EXAMPLE SJT – BACKGROUND PAGE 1 ................................................ 18 FIGURE 9. EXAMPLE SJT – BACKGROUND PAGE 2 ................................................ 19 FIGURE 10. EXAMPLE SJT PROMPTS THE STUDENT FOR THE BEST COURSE OF ACTION ....................................................................................................................... 19 FIGURE 11. EXAMPLE SJT – FEEDBACK .................................................................... 20
1
TRAINING JOINT, INTERAGENCY, INTERGOVERNMENTAL, AND MULTINATIONAL (JIIM) PARTICIPANTS FOR STABILITY OPERATIONS
INTRODUCTION
The operational environment for U.S. military forces requires the comprehension of significant amounts of new information to gain deep understanding of problem sets. Training must be relevant and should incorporate context for the new knowledge to be gained rapidly and retained well. Scenario based training can accelerate knowledge acquisition and retention.
The purpose of this project was to support the training and education process that prepares U.S. forces and their potential partners to plan and implement Stability Operations in Joint, Interagency, Intergovernmental, and Multinational (JIIM) environments.
The approach taken started with analyzing requirements for understanding Stability Operations in the JIIM environment and understanding the target audience. A set of themes was developed that reflect the high-level cognitive skills that experts use to be successful in JIIM environments. Documentation of the themes can be found in Ross, Grome, Arrastia, Schaab, Ong, and Spangler report published by the U.S. Army Research Institute in 2010.
Guided by the training requirements and training design concept, a collection of computer-based training tutorials called the Stability Operations in Joint, Interagency, Intergovernmental, and Multinational (JIIM) Environments Tutor was developed. Three versions of the tutorials were developed iteratively. Training content was expanded and refined in each new version. Each version was pilot-tested by students attending the Advanced Stability Operations course at the U.S. Army Command and General Staff College (CGSC). After each version of the tutorial was used by the class, feedback was gathered from students and instructors to identify additional improvements which were incorporated in the next version.
Traditional training systems present information and then test their recall. By contrast, the tutorials interweave the learning of FM 3-07 and other doctrine with problems and scenarios. These scenarios challenge students to apply their knowledge and cognitive skills to assess situations and generate and evaluate options in representative situations. This training approach accelerates learning by providing scenario-based learning goals and contexts for acquiring, integrating, and retaining declarative knowledge more effectively (Ross & Lussier, 1999; Schaab & Dressel, 2001). The scenarios also provide practice opportunities for applying high-level cognitive skills described by the JIIM themes.
Training effectiveness of the tutorial that taught the Interagency Management System (IMS) was evaluated by analyzing pre- and post-test scores and student responses to a reaction survey. To enable instructors and subject matter experts to create and maintain these types of training systems, an authoring tool was developed that was built on top of an existing technology.
2
METHODS Domain Analysis
A number of JIIM-related documents were reviewed to understand the issues, challenges, and lessons learned about operating in this environment. Findings were documented in a lessons learned report by Agrait and Loughran (2007). Twenty-two additional documents were reviewed which resulted in extraction of approximately 400 items of domain knowledge comprised of facts, concepts, skills, guidelines and heuristics, tasks, and challenging situations that relate to JIIM stability and support operations. This background information provided training requirements and provided details which could be included in training scenarios.
During the project, doctrine, processes, organizations, and field experiences in the area of Stability Operations evolved rapidly, and they continue to evolve. In particular, the new Stability Operations (2008) (Army Field Manual 3-07) was published, and the Interagency Management System (IMS) was evolving under the direction of the State Department’s Office of the Coordinator for Reconstruction and Stabilization (S/CRS). This manual defines Stability Operations more clearly than any prior document, and it embeds them in Comprehensive Approach that calls for regular coordination, consultation, and interaction among all actors involved including the range of JIIM participants. The manual is organized around five stability tasks to integrate activities at the tactical level (see Figure1):
1. Establish Civil Security, 2. Establish Civil Control, 3. Restore Essential Services, 4. Support to Governance and 5. Support to Economic and Infrastructure Development.
Five tasks support the Stability Sectors to create unity of effort and a vision of the desired end state across all participants in the JIIM environment. The sectors are:
1. Security, 2. Justice and Reconciliation, 3. Humanitarian Assistance and Social Well-Being, 4. Governance and Participation, and 5. Economic Stabilization and Infrastructure.
Development of Training Themes
People with experience in some aspect of JIIM environments were interviewed. The goal was to compare multiple perspectives at tactical and operations levels, particularly given the attention that has been focused on how tactical interactions have operational and strategic impacts in modern operations (see for example, Krulak, 1999). Participants in the military or associated with military organizations were volunteers, and participants from other organizations were compensated for their time as subject matter experts. Twenty-five people were interviewed:
• Seventeen military or military-related Subject Matter Experts (SMEs) (one of which was not used for analysis)
• Three African nationals, one affiliated with a non-governmental organization and two with a
3
government health organization • Five SMEs with Department of State experience
Figure 1. Stability Tasks Linked to Stability Sectors (adapted from U.S. Army FM 3-07, Stability Operations, October 2008, paragraph 2-20)
A semi-structured interview protocol based on the Critical Decision Method (CDM; Crandall, Klein, & Hoffman, 2006; Hoffman, Crandall, & Shadbolt, 1998) was used. All interviews were recorded and transcribed for analysis. The interview questions were developed to help indirectly assess the elements of expertise by gathering rich examples of performance and insights and strategies that drove performance. One assumption of the research is that as expertise develops, people generally are not able to articulate the how and why of their cognitive performance. Therefore, the analysis is grounded in the specifics of the data, but the data themselves were collected to expose the cognitive challenges and strategies inherent in situations without directly asking participants why they did things the way they did.
Design Principles
In August 2008, the team received the opportunity to work with the U.S. Army Command and General Staff College (CGSC). CGSC had developed and was delivering a new course called Advanced Stability Operations and agreed to allow us to use students in the class to pilot-test successive versions of the product. This relationship helped to focus the training requirements.
O F F E N S E
D E F E N S E
STABILITY TASKS STABILITY SECTORS
Establish Civil Security Security
Establish Civil Control Justice and Reconciliation
Restore Essential Services Humanitarian Assistance and Social Well-Being
Support to Governance Governance and Participation
Support to Economic and Infrastructure Development
Economic Stabilization and Infrastructure
FULL SPECTRUM OPERATIONS S
T A B I L I T Y
4
The challenge was figuring out how to integrate foundational knowledge along with insights into successful collaboration for planning and implementation for a diverse training audience. The potential distance learning audience for the course was comprised of students from different Services, different countries, and different government agencies, much like the composition of players who would be learning and performing together in real operations. Participants in the course were projected to include Army officers in attendance at CGSC, U.S. officers from other U.S. services, foreign officers in attendance at CGSC or currently assigned to other U.S. military organizations, U.S. Government civilians, members of non-government organizations, and students from other countries. These potential learners varied widely in their knowledge and experiences with JIIM operations.
Based on the training requirements analysis described above and the subsequent work with instructors at CGSC, the training system focused on providing students with an understanding of:
• the foundation knowledge needed to understand Stability Operations, • the high-level skills necessary to perform successfully in JIIM environments, • the interplay between tactical and operational plans and actions, • the new U.S. Army doctrine for Stability Operations (FM 3-07), and • the Interagency Management System (IMS).
The high-level objectives for the training were to enable students to:
• Understand the whole of government approach and the comprehensive approach and their challenges;
• Understand the tasks, issues, and challenges in the five sectors of Stability Operations; • Gain insight into the high-level cognitive skills that experts use to support success in JIIM
environments; and • Connect the doctrine and procedures to specific contexts to enable successful participation
in more advanced exercises and training.
It was agreed that everyone in the potential audience could benefit from participating in initial training to develop a common base of declarative knowledge, as well as insights into how to work together. The tutorial product was intended to provide the common basis that would allow learners to prepare for exercises and operations and be ready to address challenging issues with a common language and understanding of the goals of Stability Operations and the IMS.
Based on the results of the document review, interviews with SMEs, and the themes derived from these interviews, the following were developed: a general training concept, a set of design principles, and high-level learning objectives to serve as a framework for development. Specific learning objectives were developed later for each portion of the training product. The concept for this product was individual learning based on blending a constructivist approach that situates learning in a context with a Socratic method that poses questions and challenges to encourage learners to think through the context. By “constructivist approach” we mean that the learning of declarative knowledge and concepts was situated in specific contexts, requiring the students to use the knowledge as tools to solve problems and answer questions within those contexts. The choice of the constructivist approach was based on the diversity of learning levels
5
and experience in the training audience, the need to accelerate learning, and documented limitations of typical introductory training.
The goal was to accelerate entry-level learning on terminology and operations for students. It would prepare them for effective participation in class exercises for the pilot audience and also for large-scale exercises with a wide range of participants. To accelerate learning, we:
• embedded declarative knowledge in specific contexts, • provided principles of successful practice in JIIM environments (themes), and • presented challenges faced during operations for the student to work through using those
themes.
The use of operational issues illustrated the high-level cognitive challenges such as organizational and cultural clashes that can occur in JIIM environments. In general, operational issues were presented simultaneously with declarative knowledge in context. This approach allowed more advanced students to be engaged in the material as well. The presentation of situations from multiple perspectives allowed students to practice the expert ability of perspective taking. The following design principles support this training concept:
1. To support the less experienced learner, the cases are constrained, but rich enough to avoid one right answer or only one exemplar in a lesson which leads to brittle performance. Specific elements of the context were provided to the learner that supported the concepts to be learned. For example, if the tutorial focused on the security aspect of a situation, richer information about the context would be provided in this area. If humanitarian issues, for example, were also part of the situation, only enough detail to support analyzing the situation was supplied. For students who were willing and able to go further into detailed context of the situation, additional readings were provided.
2. Mental model development is supported by practicing the thought processes necessary for success in the domain (i.e., recognizing relevant information, organizing information, making sense of situations, and using information to solve problems.) Exercises and questions in each tutorial required students to analyze situations and make choices that a professional in the setting might have to make, although with more guidance as to the necessity of the analysis or choice and more structure to guide the process. For example, choosing appropriate agencies to partner with based on the problem required situation analysis and the action of choosing partners with relevant expertise.
3. Feedback allowed students to see what they have missed in situations and how more expert practitioners view the situation, as well as the consequences of concentrating on the wrong aspects of a situation. Sample expert responses were provided with an explanation of why the response suited the situation. Feedback was provided in the form of examples of expert answers and pre- and post-test. This enables students to gauge their understanding by comparing their thought processes to others and by performing checks on knowledge.
6
4. Working through “messy” situations promotes better concept integration when the learner must try to understand a situation before feedback is given. Situations did not clearly point to one right answer and required thought and choices among a range of alternatives including some that were workable, but perhaps not the most effective possible response.
5. The student focus was on understanding a mission in order to integrate concepts. Support was provided in the form of advanced organizers such as learning objectives and highlighting core concepts. Students were not left to “wander” aimlessly through a situation. Instead, they were supported with organizers to understand what concepts were being illustrated and practiced.
Development Process
The tutorials were developed in three iterations. Each version was pilot-tested during
CGSC Advanced Stability Operations course offerings in October 2008, May 2009, and November 2009. The following table summarizes the training content included in each version:
Table 1. Summary of Training Content Included in Each Version of the Tutorials
Version 1
Oct 2008 • Security in Sudan
• Governance and Justice in Haiti
• Economic Development and Humanitarian Assistance in Afghanistan
Version 2
May 2009 • Introduction to Operations in JIIM Environments
• Cross-cultural Competency Index
• Pre- and post-tests and situation judgment tests added to the Sudan tutorial
Version 3
Nov 2009 • Interagency Management System in Action!
• Pre- and post-tests for the Sudan, Haiti, Afghanistan, and IMS tutorials
After each pilot test, quantitative and qualitative reaction survey data collected from students was analyzed. Additional content and software usability improvements were identified and incorporated in the next version of the tutorials. The individual tutorials and examples of inherent design principles are described in the Findings and Discussion section.
Evaluation and Review
Participants
The research included 18 individuals ranging from E-8 to O-6 who participated in all components of this research. Five participants were students in the Advanced Stability Operations seminar at CGSC. Additional participants with relevant military backgrounds were recruited to take The Interagency Management System in Action tutorial for the TEE. These participants came from all services (Air Force, Army, US Coast Guard, Navy, and Marines) and multiple countries (United States, Kenya, Slovenia, Latvia, Korea, and El Salvador). They
7
averaged 20.2 years of military service and had been deployed all over the world (Iraq, Afghanistan, Bosnia, Saudi Arabia, Hungary, Georgia, Korea, Thailand, Australia, Turkey, Italy, UAE, Haiti, Somalia, Sudan, Kosovo, and Kuwait). Materials
The Interagency Management System in Action tutorial
• Employing the IMS
. The tutorial was designed as an introduction to the IMS. It focused on the following areas:
• Integration of the IMS with a Military headquarters • Operating as a JIIM Team • Team building and collaboration issues • Case Study activities and discussion
Knowledge Tests. A knowledge test was used to assess student declarative knowledge and the application of that knowledge with regard to this tutorial. Prior to the start of the tutorial, students completed a 25-item knowledge test to obtain a baseline of their existing knowledge of IMS, JIIM, and collaboration issues. The test items were developed based on the tutorial’s learning objectives. After completion of the tutorial, students completed the same 25-item knowledge test asking them to apply knowledge obtained during the tutorial. The knowledge tests and answers are shown at Appendix A.
Participant Reaction Survey
. Participants completed a Reaction Survey at the completion of the tutorial. The intent of the survey was to collect participants’ opinions regarding how well and in what ways the tutorial supported stated learning objectives. The participants also completed items regarding demographic information including rank, military branch, nationality, years of service, and previous deployments as part of the survey.
Procedure
Prior to data collection, the purpose of the training effectiveness evaluation was described to participants. Immediately before taking the pre-test, participants reviewed a two-page document to learn how to operate the tutorial to progress to the next screen, review previous screens, and access related documents. They were told that the tutorial required approximately two to four hours to run, depending on the individual.
Participants completed a pre-test comprised of 25 multiple choice questions. Then, they completed the tutorial. Finally, they completed a post-test comprised of the same multiple choice questions and filled out a Reaction Survey. The results of this training effectiveness evaluation are reported in the Findings and Discussion section.
In addition to the primary testing at CGSC, some or all of the product components were reviewed during the final stage of development by organizations that were considering using the tutorials. These included:
• instructors and students at Joint Forces Staff College and • members of the Office of the Secretary of Defense (OSD) Personnel and Readiness.
This department focuses on Interagency Training integration and works with the S/CRS in the development of their civilian deployable capability. Reviewers from
8
OSD had previously attended the two week S/CRS-sponsored IMS Foundations course, and they reviewed the tutorial Interagency Management System in Action!
Authoring Tool Development and Usability Assessment
To support the development of the tutorials, we enhanced an existing authoring tool called Task Guide. An overview of this tool can be found in Appendix B. After version 1 of the tutorials was developed, additional capabilities were added to the authoring tool to simplify authoring. These included:
• A graphical editor for specifying formatted text and input controls (e.g., text fields, check boxes) using a third-party commercial software component. When creating version 1, it was necessary to enter raw Hypertext Markup Language (HTML) tags to specify the formatting and input controls.
• Simplified support for authoring multiple choice items with response-specific hints and feedback. Using this support, authors could specify all aspects of a multiple choice item such as its question (prompt), possible options, correct options, and response-specific hints and feedback for each correct and incorrect option. Once this information was entered, the author could quickly insert screens that pose questions, prompt for selections, and provide hints and feedback at the desired locations in the tutorial.
• Generation and conditional inclusion of text within a screen. This feature enables tutorial authors to design screens that include logic that generates text or selects among alternate blocks of text based on the student’s previous interactions and other information. These features enhance Task Guide’s ability to support adaptive instruction.
The tool was modified to improve the student experience by reducing the amount of scripting that was needed to author certain kinds of tutorial logic such as saving each student’s answers and scores in files. The following documents that describe how to operate Task Guide and maintain the tutorials were written or revised:
• Task Guide Users’ Guide • Task Guide Authors’ Guide • Maintaining the Stability Operations Tutorials
To assess the usability of the authoring tool, a former instructor at Joint Forces Staff College reviewed the Task Guide manuals. Then, during a WebEx web conference, developers demonstrated how to operate the authoring tool. The instructor then applied this knowledge to change portions of an existing tutorial. The instructor then provided feedback by completing a written questionnaire (presented in Appendix C).
Tutorials
This section describes the training content developed and included in the three versions of the tutorials.
Tutorial version 1 – October 2008
This version of the training system included three tutorials that illustrated the concepts of the five stability sectors and enabled the learner to work at integrating the concepts in context.
9
Several questions or dilemmas were presented to the learner throughout each tutorial to engage them, help them to become familiar with the basic doctrine, and support practicing the thought patterns needed to plan and conduct operations. Examples of expert points of view for each challenge were available once the student had an opportunity to think through the issues presented by the tutorials. The tutorials provided quick access to reference materials via hyperlinks to deepen understanding on various aspects of the problem or the context. In each tutorial, a general setting and knowledge of the type of task were provided, followed by presentation of a specific set of conditions and issues for the learner to resolve. Some of the eight JIIM themes were woven into each tutorial to support practice of these high-level skills. For example, the learner might be prompted to take the perspective of other players to answer a question.
Security in Sudan.
The Sudan situation involved the United Nations (UN) experiencing problems enforcing the Comprehensive Peace Agreement in 2007. The tutorial was designed to help the learner understand the variety of participants in this type of operation, understand the effects of cultural competence, understand the coordination necessary, make assessments of the security situation, and evaluate the different participants along several dimensions.
This tutorial addressed theme 1 (Understand the situation within its historical, regional, and cultural context) and theme 2 (Understand the other participants).
Governance and justice in Haiti.
The Haiti tutorial was based on an intervention in 1994 and required the learner to conduct a mission analysis by examining tasks from multiple perspectives and their differing priorities (Government of Haiti led by President Aristide, United Nations, and the U.S.), assessing risks and resources, and assessing stakeholder agendas.
This tutorial addressed JIIM theme 3 (Shift perspective) and theme 4 (Establish and maintain common ground).
10
Humanitarian assistance and economic development in Afghanistan.
The Afghanistan tutorial was based on a hypothetical natural disaster in which the student coordinates integration of civilian reserve personnel into Provincial Reconstruction Teams (PRTs) while simultaneously seeking to gain information about threat activities in the area.
This tutorial addressed theme 5 (Build capability to affect the situation).
Tutorial version 2 – May 2009
For version 2, two activities were added: • Introduction to Operations in JIIM Environments • Cross-Cultural Competence Inventory (3CI)
Introduction to operations in JIIM environments.
The range and variety of potential learners was the motivation for adding the Introduction to Operations in JIIM Environments module. The introduction was designed to insure that all learners understood the wide range of missions and tasks that could fall under Stability Operations, as well as the challenges and issues inherent in the operations. The introduction provided information on the concepts of the Whole of Government Approach and the Comprehensive Approach, as well as describing the Stability Operations tasks to insure a common understanding of the basics of doctrine. The role of the
US military as an actor and not a leader in many situations was also addressed.
11
Cross-cultural competence inventory (3CI).
The Cross-Cultural Competence Inventory (3CI) was added to sensitize the student to the role of cultural interactions in JIIM operations. 3CI is an 85- item self-report assessment tool under development by the Cognitive Performance Group. An intermediate version of the inventory was added to make the tutorial more engaging to students and to cause students to reflect on their attitudes and work style and how these elements impact operations. Students complete the self-assessment items using a
Likert scale from 1-6, where 1 meant “Strongly disagree” and 6 meant “Strongly agree.” Feedback on each of six scales is given in the form of short paragraphs based on whether the scale score is above, about equal to, or below the mean, calculated from a sample of 641 military members across the Services. Feedback is general and statements are provided for each of the six scale scores within the 3CI. For example, feedback for the Willingness to Engage scale is as follows when the score is below average: “You scored somewhat lower than average on Willingness to Engage. People who score lower may need to work harder on getting to know the different players in the JIIM environment. This is an important skill to develop as it will assist you in establishing relationships that allow you to understand others’ perspectives and goals.”The student is also presented with the average scale score and their own scale score.
Tutorial version 3 – November 2009
For the third version, a tutorial called The Interagency Management System in Action! was added to teach the IMS. We also added pre- and post-tests to the Sudan, Haiti, Afghanistan, and IMS tutorials.
The interagency management system in action!
The Interagency Management System in Action! tutorial was designed as an introduction to the IMS. It enabled students to learn about the IMS structure, governance, and Stability Operations doctrine and to apply this knowledge in a generic, hypothetical situation. This tutorial focused on the IMS and when and how it operates, rather than on the geography and politics of a particular region. Operations in JIIM environments will require new and different ways of thinking and collaborating. This environment will require a shared understanding of
both problems and potential solutions offered by each JIIM partner. This tutorial addressed the
12
challenges and structures for collaboration, and it was designed to develop critical thinking and reasoning skills needed to conduct successful Stability Operations. Tutorial interactions
The expected benefit of the tutorial approach is that problems challenge students to apply what they already know. The scenarios help students with diverse backgrounds and proficiency levels learn the declarative knowledge more effectively by providing specific situations that cause students to identify the important issues and questions. In this way, the scenarios provide students with specific questions that motivate them to learn the doctrine and country-specific background information in an active, goal-directed way. For example, a scenario the Security in Sudan tutorial discusses the Integration Planning Cell (IPC) and prompts the student to select skills that would be useful to include in the IPC, as shown in Figure 2. By contrast, traditional training often requires students to passively absorb large amounts of information up front before applying it to solve problems.
Figure 2. Security in Sudan tutorial prompts for useful skills to include in the Integration Planning Cell
13
Before providing feedback on the student’s selections, the tutorial asks the student to identify the key drivers of conflict as shown in Figure 3. This screen refers the student to relevant country-specific information (United Nations analysis of Sudan) and doctrine (a chapter from FM3-07 that discusses Interagency Conflict Assessment Framework) accessible via hyperlinks. In this manner, the tutorial poses a specific question that:
• guides the student’s thinking with respect to the original question about useful IPC skills • motivates and focuses the student’s acquisition of knowledge about Sudan and conflict
assessment.
Figure 3. Security in Sudan tutorial poses a question to guide the student’s thinking and to motivate the acquisition of country-specific and doctrinal knowledge
14
When the tutorial asks the student a free-text question, the tutorial does not evaluate the student’s answer because it is technically difficult for software to interpret free text reliably. However, the tutorial presents a reference answer that students can compare with their own answers, as shown in Figure 4.
Figure 4. Security in Sudan tutorial provides a reference (or expert) answer that students can compare with their own answers.
15
This question about key drivers of the conflict serves as a hint that helps the student reflect on which IPC skills are important. The tutorial then provides a second opportunity for the student to select important IPC skills, as shown in Figure 5.
Figure 5. Security in Sudan tutorial re-prompts the student to select useful IPC skills.
16
Finally, the tutorial provides feedback by showing the preferred answer and the student’s answer, along with rationale. As shown in Figure 6, the student’s selected choices are checked, and the preferred choices are displayed in blue.
Figure 6. Security in Sudan tutorial provides feedback and rationale for a multiple choice question.
Some of the tutorials provided response-specific hints after the student has answered a multiple choice question. For example, the tutorial might point out relevant information for each missing or incorrect choice selected by the student. After presenting the hint, the tutorial provides the student with an opportunity to change their answer to the original question. Finally, the tutorial provides feedback by displaying the student’s selections, the preferred selections, rationale for the preferred selections, and reasons why incorrect choices selected by the student, if any, are not preferred.
17
Knowledge checks
To ensure understanding, examples and numerous items were used as “knowledge checks” before the more complex tutorials were accessed. The themes were also introduced as skills to be developed for success in the JIIM environment to sensitize the learner to the fact that the elements of success were more complex than understanding doctrine and processes. The tutorial presented a series of situations and prompted students to identify relevant themes. An example situation and prompt is shown in Figure 7.
Figure 7. Example situations prompt students to identify relevant themes Situation judgment tests
Situation Judgment Tests (SJTs) were added to the Sudan tutorial and were labeled as Practical Exercises. These practical exercises used a rich context and allowed the students to make complex situation assessments without entering free-form texts, because SJTs use a multiple choice format. SJTs are aimed at assessing knowledge of concepts, integration of multiple concepts and application of that knowledge to realistic dilemmas.
To develop an SJT, the author must understand typical dilemmas and issues faced in the environment and domain of practice. Drawing on prior understanding of conflicts and assessments at the tactical and operational level of Stability Operations, rich descriptions were developed of situations and a statement of the best response possible that took into account all
18
the variables outlined in the background description and potential second and third order consequences. Then up to three other statements that describe plausible responses to the situation were developed. For each statement, feedback was developed on what was good about the response and what could cause potential problems if the response was selected as the desired action. Responses to SJT items can also be used to sum up performance to a user at the end of a module of tutorial. Summing up the level of performance could provide more diagnostic feedback. It can also be part of an ongoing, continual assessment to see if the user grasps concepts and integrates concepts more effectively as the training progresses. To provide the most diagnostic feedback, the developer must identify domain knowledge and concepts or themes for the domain to each SJT and each response during the authoring process. Feedback can then indicate how successful the choices were across SJTs and link the choices to how well particular knowledge was applied. Figure 8 through Figure 11 show the background information, prompt, and feedback for an example SJT.
Figure 8. Example SJT – background page 1
19
Figure 9 . Example SJT – background page 2
Figure 10. Example SJT prompts the student for the best course of action.
20
Figure 11 Example SJT – feedback
FINDINGS AND DISCUSSION
Training Effectiveness Evaluation The goal of the evaluation was to identify the effectiveness of the Interagency Management System in Action! tutorial by analyzing the differences between the pre- and post-test scores. Data was examined at the .05 level of statistical significance. Pre- and post-test results
The average number correct on the multiple choice pre-test was 11.6 out of 25 questions (46.4%), and the average number correct on the post-test was 17.8 of 25 (71.2%), a difference that was significant using an independent samples t-test (p<0.05).
An item analysis was performed to determine which of the multiple choice test items best discriminated the performance within the group. Student scores for 11 of the 25 items showed significant difference between the pre- and post-tests (p<.05). Appendix A contains the full knowledge test with correct answers indicated and values for each item indicating those that showed a significant difference in discriminating between the pre- and post-tests.
Reaction survey results
The Reaction Survey responses were analyzed by looking at the mean score on each of the questionnaire items. The scoring for all items was based on a five-point scale:
1 Strongly Disagree 2 Disagree 3 Neutral 4 Agree 5 Strongly Agree
21
The Reaction Survey was separated into three categories. The first category focused on the participants’ overall attitudes toward the training. The most positive results were participants agreeing that:
• The Interagency Management System in Action! tutorial enhanced my understanding of the concepts of the course (mean=4.05)
• The tutorial helped me to look at Interagency operations from several perspectives (mean=4.00)
• The technology was easy to download and use (mean=3.95)
The most negative result was: • The tutorial took too long to complete (mean=3.45)
The results of this section of the survey are displayed in Table 2.
Table 2, Reaction Survey Items and Means for Overall Attitude Toward the Training.
Reaction Survey Question Mean
1. The “IMS in Action” tutorial enhanced my understanding of the concepts of the course.
4.05
2. The tutorial was often boring. 2.55
3. The tutorial was a valuable learning experience for me. 3.90
4. The tutorial was easy to use. 3.55
5. I would like to participate in more tutorials like this in the future. 3.55
6. The tutorial helped me to look at interagency operations from several perspectives.
4.00
7. The material presented in the tutorial was clear and easy to understand. 3.60
8. The tutorial took too long to complete. 3.45
9. The technology was easy to download and use. 3.95
10. Using the tutorial has improved my ability to understand other organizations and their functions.
3.85
11. I am better able to understand the desired role of the US military’s role in Stability Operations.
3.70
12. I feel better able to implement a Whole of Government approach to Stability Operations.
3.70
22
The second part of the Reaction Survey focused on how well the tutorial supported various learning objectives. The scoring for each learning objective was based on a five-point scale where a “1” signified that they did not feel that the training supported the learning objective and a “5” illustrated strong agreement that it supported the learning objective. Participant reactions toward the tutorial teaching the learning objectives ranged between 3.35 and 3.95.
The most positive results were agreements by participants that the tutorial supported the following learning objectives:
• Identify the wide array of actors within Stability Operations in the U.S. Government and larger Stability Operations community (mean = 3.95),
• Explain the Interagency Management System as a method for facilitating the WGA approach (mean=3.95), and
• Explain the U.S. Whole of Government Approach (WGA) to Stability Operations (mean=3.90).
Table 3 shows the items and the response means for this section of the survey.
Table 3. Reaction Survey Mean Ratings for How Well the IMS Tutorial Supported Learning Objectives.
Learning Objective Mean
13. Identify the wide array of actors within Stability Operations in the U.S. Government and larger Stability Operations community.
3.95
14. Explain the role of the Military in accordance with the WGA. 3.55
15. Explain the Interagency Management System as a method for facilitating the WGA approach.
3.95
16. Explain the U.S. Whole of Government Approach (WGA) to Stability Operations. 3.90
17. Describe the IPC portion of the Interagency Management System as a method for facilitating the WGA.
3.80
18. Differentiate the capabilities of the different Interagency organizations in Stability Operations.
3.60
19. Describe how the WGA and DoD organizations align both structure and processes when the IMS is initiated.
3.70
20. Differentiate the capabilities of the different Interagency organizations in Stability Operations.
3.70
21. Describe the Interagency Management System (ACT specifically) as a method for facilitating the WGA at the JTF level.
3.70
22. Define how the U.S. military can support goals of other U.S. government agencies in Stability Operations.
3.70
23. Describe how the WGA and DoD organizations align both structure and processes when the IMS is initiated.
3.70
24. Compare different methods of solving the same problem. 3.35
23
25. Differentiate the capabilities of the different players in Stability Operations. 3.55
26. Demonstrate an understanding of the perspectives of others around a single situation or decision.
3.45
27. Identify common goals among players in Stability Operations. 3.60
28. Identify typical sources of friction among different players in Stability Operations. 3.35
The third section containing written feedback from participants showed that the tutorial was a useful learning experience and that it could be improved in some areas, as shown below in Table 4.
Table 4. Reaction Survey Comments about Strengths and Weaknesses of the Tutorial
Positive Feedback Places for Improvement • Good in-depth material and introduction
to IMS • Forces student to be able to apply
material
• Material too dense/difficult/too time consuming • Too many abbreviations/acronyms not spelled out • Software awkward/Not user friendly (e.g., No
ability to move around within tutorial or go back without losing data).
• Difficulty getting the software to install • Add video/audio/diagrams/pictures/interactive
component. Limitations of the findings
The participants in this evaluation were motivated to participate in this tutorial as indicated by the Reaction Survey data. Many researchers have identified the importance of student motivation to learn as a key factor in any effective training program (Baxter, 2003; Ilgen & Klein, 1989; Locke & Latham, 1984; Luthans & Stajkovic, 1999). The motivational characteristic of this type of tutorial is clearly a factor in its favor compared to other training media. A number of reasons have been identified which help digital-based training systems achieve their motivating effects. These include clear cut goals, fast pace, immediate feedback, and variable or increasing levels of challenge (Chaffin, Maxwell, & Thompson, 1982).
Although a test-retest without the tutorial training would have acted as a control group to
measure any gains in post-test scores simply as a function of practice taking the test, there were not enough students available for this activity.
Participant acceptance of training is an important criterion and one that is heavily relied upon in training evaluation research. A caution, however, is the increasingly well-documented finding that training interventions that produce difficulties for trainees during acquisition, and that may frustrate trainees, may actually produce better learning as evidenced by transfer and retention. Baddeley and Longman (1978), for example, found that training conditions most preferred by students resulted in less learning. A conclusion from Schmidt and Bjork’s (1992) review of this
Table 3. (continued)
24
literature is that valuable training interventions may go undetected because rarely do evaluations assess transfer and retention. Thus, a caution is that the present evaluation results apply to baseline knowledge acquisition only and are not indicative of actual transfer to field or exercise performance which was not assessed in this project. Authoring tool usability assessment
A former instructor at Joint Forces Staff College reviewed the Task Guide documentation, received a demonstration of the tool, and made several practical changes to the tutorial. Based on this experience, he concluded that the tool was useful for creating effective instructional content and that one hour of preparatory document review and two and a half hours of instruction were sufficient to enable him to make simple changes to an existing tutorial. He estimated that three additional hours of document review and instruction would enable him to make major changes; including creating a tutorial from scratch, assuming that the instructional design of the tutorials had already been carried out. Pre-requisites for successful use of the tool include: attention to detail, advanced preparation of changes, detailed knowledge of the subject matter and instructional plan, documentation of changes and version control, and a basic level of computer familiarity common among instructors who utilize education technology systems/networks. His response to the questionnaire is presented in Appendix C.
CONCLUSIONS AND RECOMMENDATIONS
This collection of training tutorials is a training product which provides a baseline understanding for a potentially large and diverse training audience for an area of practice that is vital to operations but not yet widely integrated in our education and training systems. Few opportunities exist to share this information among the variety of participants who must engage in these operations. The doctrine and knowledge being produced to support Stability Operations, the IMS, and non-kinetic operations is, in general, scattered and diverse. The amount and type of reference material already in the tool provides a useful reference library and place for different users or user groups to potentially store additional material as they use the product. This tool fills a need that no other distance learning product is currently addressing, and it does so in a manner that supports the range of learning from knowledge acquisition of basic concepts to the integration and application of those concepts. In addition, the tool provides relevant processes and procedures, as well as insight into the high-level cognitive skills needed for successful application in an operational environment where collaboration is at its most challenging. The specification of learning objectives for each module within each tutorial supports the instructional planner in integrating the product into a larger training program. The authoring tool that is available with the product also allows for scenarios and assessment items to be modified or added as learning objectives change and new examples are generated from practice.
Transition to Operational Use
The tutorials have been selected for use by the Joint, Interagency, and Multinational Planner's Course (JIMPC), offered at Joint Forces Staff College. The IMS Tutorial has been reviewed by members of the Office of the Secretary of Defense (OSD) Personnel and Readiness and the Department of State. It is under consideration by US JFCOM to educate Combatant Commander and Joint Task Force staffs on the Interagency Management System who cannot
25
attend the S/CRS Foundations course and must rely on distance learning methods. The tutorials are also being evaluated for wider use by other schools at Joint Forces Staff College.
Recommendations
In general, digital scenario-based training is most effective when it is embedded in a larger training program containing a number of other training elements such as classroom and live training (Prensky, 2001). Integration should include:
• Identification of specific course training objectives that can be met with the tutorial. • Development of scenarios that allow deliberate practice with respect to the learning
objectives. • Development of performance assessment tools so that achievement of learning objectives
can be measured. • After action review to provide feedback to trainees regarding how well learning
objectives were achieved.
The feedback provided in the tutorials presents ideas for further performance assessment and after action review in a more comprehensive training program. Feedback provided by additional user groups can be incorporated to improve the tutorials, and the results of different uses of the tutorials, such as standalone use or integration into a larger curriculum, should be documented.
The addition of more SJTs can serve as learning tools to practice using concepts but can also be used as an ongoing assessment. Summing up findings across SJTs can also provide an after-action review (AAR) capability. The SJTs in the current tool can serve as examples for instructors who want to learn to develop these short vignette-based assessments using current experiences and historical cases. More sophisticated Socratic dialog techniques could be added to these tutorials and to new tutorials that take greater advantage of Task Guide’s ability to present information, provide hints, and pose questions adaptively to guide the student’s thought processes.
In conclusion, to successfully use the product, adopters must integrate it with their objectives and determine how they want to use the tool for best effect. The entire tool can be used to establish a baseline for a heterogeneous training audience to enhance later engagement in live exercises. The tool can also be used as part of a larger curriculum. Subsets of the tool can be used to introduce concepts. For example, adopters could use only the Interagency Management System in Action! tutorial as an introduction to the IMS to meet some objectives of a larger curriculum.
26
27
REFERENCES
Agrait, R. & Loughran, J. J. (2007). Lessons Learned from Joint, Interagency, International and Multinational (JIIM) Workshops and Papers. Vienna, VA: ThoughtLink, Inc.
Baddeley, A.D., & Longman, D.A. (1978). The influence of length and frequency of training
session on the rate of learning to type, Ergonomics, 21, 627-635. Baxter, H. C. (2003). A comparison of diversity training methods for incorporating people with
disabilities into the workplace. Bloomington, IN: Indiana University. Chaffin, J.D., Maxwell, B., & Thompson, B. (1982). ARC-ED Curriculum: the application of
video game formats to educational software. Exceptional Children, 49, 173-178. Crandall, B., Klein, G., & Hoffman, R. R. (2006). Working minds: A practitioner’s guide to
cognitive task analysis. Cambridge, MA: The MIT Press. Hoffman, R. R., Crandall, B. W., & Shadbolt, N. R. (1998). Use of the critical decision method
to elicit expert knowledge: A case study in cognitive task analysis methodology. Human Factors, 40(2), 254-276.
Ilgen, D. R., & Klein, H. G. (1989). Organizational Behavior. Annual Review of Psychology, 10,
327-351. Krulak, C.C. (1999, January). The Strategic Corporal: Leadership in the Three Block War.
Retrieved from the Web October 2, 2009. Locke, E. A., & Latham, G. P. (1984). A theory of goal setting and task performance.
Englewood Cliffs, NJ: Prentice Hall. Luthans, F., & Stajkovic, A. D. (1999). Reinforce for performance: The need to go beyond pay
and even rewards. The Academy of Management Executive, 13, 49-57. Prensky, M. (2001). Digital Game Based Training. New York: McGraw-Hill. Ross, K.G., Arrastia, M. C., Grome, A., Schaab, B. B., Ong, J., Spangler, D., (2009).
Development and Validation of Training Themes for Joint, Interagency, Intergovernmental and Multinational (JIIM) Operations. Proceedings of the Industry/Interservice, Training, Simulation & Education Conference. Orlando, FL.
Ross, K.G., Grome, A., Arrastia, M. C., Ong, J., Spangler, D., & Schaab, B. B. (2010).
Development of training themes for Joint, Interagency, Intergovernmental and Multinational (JIIM) Operations, (Research Product 2011-04). Arlington, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
28
Ross, K. G., & Lussier, J. W. (1999). A training solution for adaptive battlefield performance. Proceedings of the 1999 Interservice/Industry Training, Simulation, and Education Conference, Orlando, FL.
Schaab, B. B., & Dressel, J. D. (2001). Training for adaptability and transfer on digital system,1963.
(Research Report 1782). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
Schmidt, R.A., & Bjork, R.A. (1992). New conceptualizations of practice: Common principles
in three paradigms suggest new concepts for training. Psychological Science, 3, 207-217. U.S. Department of the Army. (2008). FM 3-07. Stability Operations. Washington, DC:
Headquarters
A-1
APPENDIX A
IMS TUTORIAL KNOWLEDGE TEST ITEMS AND SIGNIFICANCE VALUES FOR DISCRIMINATING PRE- AND POST-TEST SCORES
A-2
Question Correct
Answer P Value
1. The purpose of an integration planning cell is a. To support civil-military communication and harmonize
civilian military planning b. To create a USG R and S Operations Plan c. To create a USG R and S Implementation Plan d. All of the above e. None of the above
a 0.002*
2. On an IPC, the State Department works on all of the following except: a. Planning, Operations, and Management b. Criminal Justice and Policing c. Diplomacy and Governance d. Essential Services e. Diplomatic Security
d 0.016*
3. Criminal Justice and policing is handled by a. State Department b. Justice Department c. Homeland Security d. USAID e. All of the Above
e 0.124
4. The Interagency Planning Framework used by the US Government consists of a. Storming, Forming, Norming, Performing b. Assessment, Execution, Evaluation c. Assessment, Strategy Development, Execution d. Policy Formation, Strategy Development, Implementation
Planning e. None of the above
d 0.029*
5. The elements of a plan that are necessary to achieve the Mission Conflict Transformation Goal are called: a. Overarching Policy Goals b. Major Mission Elements c. Essential Tasks d. Planning Template e. Implementation Planning
b 0.000*
6. An example of a Major Mission Element is: a. Immediate humanitarian needs addressed b. Hydrocarbons c. Security Assistance
e 0.003*
A-3
d. Regional Disputes e. All of the Above
7. What is the ACT relationship between the US Embassy and the
JTF a. ACT works with the JTF, but not with the US Embassy b. ACT works with the US Embassy, but not with the JTF c. ACT splits into 2 sections: one to work with the US Embassy
and one to work with the JTF d. ACT splits into 3 sections: One to work with each the US
Embassy, JTF, and Host Nation e. None of the above
c 0.171
8. Which of the following is false regarding the ACT relationship to the US Embassy and the JTF? a. ACT participating in the IMS will coordinate with many
different actors and entities – some of whom may not be operating under the system.
b. These other elements may include the host country (public and private sector) and its military, coalition militaries, and NGOs.
c. Within the IMS, the Joint Force Commander will have reoccurring coordination and interaction with members representing the IPC, ACT, and if deployed, the FACT.
d. Coordination with the CRSG will likely occur through guidance and direction received from OSD, Joint Staff, and the GCC using existing procedures
e. It will be important for the JTF Commander and staff to know and understand the mission and purpose of each ACT member agency so coordination requirements can be better supported.
a 0.418
9. As you transition to stability and reconstruction activities who is responsible for the execution a. JTF Commander b. Joint and Interagency Responsibility c. IPC d. ACT e. None of the Above
b 0.169
10. The US government has the developed an interagency planning and coordination framework for Stability Operations called: a. Comprehensive approach b. Joint Forces Command c. Interagency Management System d. Civilian Response Forces e. None of the above
c 0.554
A-4
11. Smart power refers to: a. Cyber operations b. A means to integrate Joint Interagency and Multinational
capabilities c. A means by which the US State Depart leverages diplomatic power d. The full range of diplomatic, economic, military, political, legal
and cultural tools available to the US government e. A National Security Objective to bring coalition partners together to
conduct Stability Operations.
d 0.124
12. Which of the following is likely to trigger the activation of the IMS a. Little to no US military involvement b. Low mortality rates in a conflict c. Large scale displacement of people d. No threat to US civilians e. Decrease of USG funded civilian programs
c 0.058
13. According to IMS triggering guidance in the case of eminent crisis with Reconstruction and stability or conflict implications, what senior officials can trigger whole of government planning? a. The Coordinator for Reconstruction and Stability b. The Secretary of State c. The Secretary of Defense d. A & B e. B & C f. All of the above
e 0.291
14. The IMS is intended to support a. Integrated planning processes b. Joint interagency field deployments c. Joint civil-military operations d. Shared communications and information management e. All of the above f. A & D
e 0.476
15. The IMS consist of three functional levels: a. The Country Reconstruction & Stabilization Group (CRSG), The
Integration Planning Cell (IPC), The Advanced Civilian Team (ACT)
b. The Department of State, The Country Reconstruction & Stabilization Group (CRSG), Advanced Civilian Teams (ACT)
c. The Advanced Civilian Team (ACT), Field Advanced Civilian Teams (FACT), The Integration Planning Cell (IPC)
d. The Secretary of State, The Secretary of Defense and the National Security Council
e. The Department of State Coordinator for Reconstruction and Stability, The US Ambassador in the affected country, the Country Team.
a 0.136
16. The IMS is not likely to be used when a. Highly complex crisis or operations which are national or securities
priorities b. There is widespread instability c. There is a humanitarian crises that are handled through the current
Washington DC and US Embassy based system *** d. A situation is likely to require military operations e. A foreign state fails or falls into crisis
c 0.003*
A-5
17. The Integration Planning Cell (IPC) deploys to and supports a. The Department of State b. The Country Reconstruction & Stabilization Group (CRSG) c. The US Embassy in the affected country d. The Joint Task Force e. The Geographic Combatant Command
e 0.003*
18. The Advanced Civilian Team (ACT) deploys to and supports a. The Department of State b. The Country Reconstruction & Stabilization Group (CRSG) c. The Geographic Combatant Command d. The Joint Task Force
d 0.008*
19. The Field Advanced Civilian Teams (FACT) deploys to and supports a. The Department of State b. The Country Reconstruction & Stabilization Group (CRSG) c. The US Embassy in the affected country d. The Joint Task Force e. The Geographic Combatant Command
d 0.082
20. The Field Advanced Civilian Teams (FACT)will locate In the provinces outlying area and operate under the direct guidance of the Integration Planning Cell (IPC) a. True b. False
b 0.735
21. The military representation on the Country Reconstruction & Stabilization Group (CRSG) comes from
i. The Department of Defense ii. The US Embassy in the affected country
iii. The Joint Task Force iv. The Geographic Combatant Command v. Joint Staff
a. i & v b. i & ii c. iii& iv & v d. i & iv & v e. all of the above
d 0.011*
22. Which one of the following is not usually part of the IMS a. Department of Justice b. Department of Defense c. Department of Treasury d. Department of the Interior e. Department of Education f. Corps of Engineers
d 0.728
23. What agency is responsible for diplomatic Security a. Department of Justice b. Department of Defense c. Department of Treasury d. Department of the Interior e. Department of Homeland Security f. Department of State
f 0.009*
A-6
24. The Interagency Conflict Assessment Framework (ICAF) is: a. The Army version of interagency planning b. A theoretical framework that assigns conflict resolution values to
countries in crisis c. Used to prevent violent conflicts d. A tool to identify causes of instability and to develop activities to
diminish and mitigate them e. A tactical planning process used by the other government agencies
to interpret military campaign planning into civilian terms
d 0.700
25. The Integration Planning Cell (IPC) leader is a. On a peer level with the Geographic Combatant Command
Commander b. Subordinate to the Geographic Combatant Command Commander c. Senior to the Geographic Combatant Command Commander d. Attached to the Geographic Combatant Command Commander e. None of the above
a 0.000*
B-1
APPENDIX B
TASK GUIDE AUTHORING TOOL OVERVIEW
B-2
The Task Guide software system enables organizations to develop sophisticated applications called procedures that retrieve, compute, update, display, and accept information using a step-by-step wizard-like user interface. Task Guide provides a graphical authoring tool that enables authors to create these procedures quickly and easily, without programming. Powerful authoring capabilities enable you to create adaptive procedures that determine which information to present and which questions to ask, depending upon the situation. Task Guide is modular and extensible: it can be embedded within a larger Java applications or applet, and it can embed Java user interface components and libraries.
Task Guide procedures can be used to implement diverse types of applications such as: • Training tutorials that present information to students, pose questions or problems, and
provide tailored hints and feedback. Student performance can be tracked in files or by a SCORM learning management system.
• Debriefing systems that analyze and discuss the student’s perceptions, decisions, and actions during a simulation-based training or assessment activity. This discussion format enables the software to refine its assessment of the student’s proficiencies, provide instructional feedback, and guide reflection.
• Performance support systems such as task aids and decision support systems that guide users through analysis, decision-making, and execution tasks, step-by-step.
• Training simulations such as branching scenarios that present the current situation in each screen, prompt the student for decision or actions, and branch to the appropriate next situation.
• Adaptive questionnaires that determine which questions to ask based on previous answers and other available data.
Flexible branching, looping, and calculation logic enables Task Guide to select the appropriate next screen and generate its contents dynamically, based on the user’s inputs and the values of the procedure’s variables. This makes it possible to create flexible, dialog-like interactions that adapt to the situation and user. For example, Task Guide training tutorials can select or generate different hints, feedback, and follow-up questions based on each student’s correct or incorrect answers to previous questions and, optionally, other information about the student. Task Guide training simulations can branch to the next situation or present possible choices based on the student’s previous decisions and the values of procedure’s variables used to store the simulation state. Task Guide debriefing systems can point out noteworthy instances of student performance, assert relevant facts, and ask probing questions about the student’s experience and perspective. After the student responds, the debriefing systems can delve more deeply with follow-up facts and questions. Task Guide performance support systems can filter, calculate, and display the information or possible options that are relevant to the current situation more succinctly, so users can assess situations, make decisions, or execute tasks more accurately and efficiently. A single Task Guide procedure can even combine different types of interactions. For example, a Task Guide procedure could provide training by interleaving tutorial interactions, branching scenarios, and debriefing.
B-3
Task Guide Training Tutorials
A Task Guide training tutorial displays a series of screens that: • Present direct (didactic) instruction to students, • Pose questions and accept and evaluate each student’s response, • Provide hints and feedback on each student’s responses, • Branch to the appropriate next screen based on each student’s response, and • Tracks the student’s responses and scores using files or a SCORM learning management
system.
Task Guide procedures can use response-specific hints, feedback, and branching to provide instruction that adapts to the student’s background and responses during the tutorial. Task Guide procedures can pose two types of questions:
• Text input questions that prompt the student for free-text responses, and • Multiple choice questions that prompt the student to select one or more choices.
Task Guide provides specialized support for authoring multiple choice items that include response-specific hints and feedback.
Task Guide Software Suite
The Task Guide software suite provides the following software programs: • Task Guide Procedure Editor - a Java application used by authors to create Task Guide
procedures, • Task Guide Procedure Execution Tool - a Java application used by students or end users to
run Task Guide procedures, and • Task Guide Procedure Execution Applet - a Java applet that runs within Java-enabled web
browsers to run Task Guide procedures. This applet is invoked by embedding an applet tag within a Hypertext Markup Language (HTML) web page.
Figure 12. Authors create procedures using the Procedure Editor. Users run procedures using the Procedure Execution Tool
Task Guide Procedure Editor
Task Guide Procedure Execution Tool
Procedure author Procedures Operator
B-4
Task Guide Procedures
Task Guide procedures are comprised of step nodes and group nodes. Step nodes specify the content and format of each screen, along with optional calculations that run before or after each screen are displayed. Group nodes contain step nodes and lower-level group nodes. Group nodes organize steps within a hierarchy, similar to the way computer folders organize files. The different types of steps and groups are shown below.
Interactive Invisible Simple Group Simple Step Branching
Group
Exit Step Looping Group Conditional Branches Step
Each simple step presents information to the user and can prompt the user for input. An exit step specifies an exit condition that, if true, instructs Task Guide to stop executing the current group. The conditional branches step specifies many possible steps to execute next, depending upon the values of certain variables. A simple group organizes related steps and subgroups. All of the steps and sub-groups contained within a simple group are executed in sequence. A branching group specifies a Boolean (true/false) condition that determines whether the steps in the group should be executed or skipped. A looping group executes its steps repeatedly while its test condition is true.
A step’s instructions can contain input controls such as text fields, check boxes, radio buttons, and selection lists that prompt the user to enter data and decisions. Task Guide stores user input values in variables, so they can be used within the calculations and test conditions of this step or downstream steps. Task Guide provides special support for specifying multiple choice questions with response-specific hints and feedback. Task Guide steps can present verifications that tell the user how to confirm that the step was completed successfully. Optional notes communicate information, cautions, and warnings. Step instructions can also embed arbitrary Java graphical user interface objects that provide specialized information displays or user-system interactions.
Steps can contain calculations that evaluate expressions and save these values in variables. These expressions can contain constant values; variables; math, text, and logical operations; and calls to Java methods. The saved variable values can be used by branching logic to control which downstream steps are executed and which are skipped. They can also be used within calculations in downstream steps to send/receive data to/from other systems and databases, analyze and interpret this data, recommend actions to be taken by the user, or select and execute actions automatically. Pre-calculations execute at the beginning of each step before the step’s instructions are displayed to the user. They are useful for retrieving and computing data or text so they can be embedded within dynamic instructions. Post-calculations execute at the end of the step, after the user has followed the step’s instructions, entered data into input fields, and pressed the next step button to indicate completion. They are useful for interpreting, processing, saving, or acting upon the user’s inputs.
B-5
Task Guide Authoring Tool
Authors use the Task Guide Authoring Tool to create procedures. A Task Guide procedure encodes step-by-step instructions and execution logic as a list of steps, organized within groups and subgroups in a hierarchy. Each step presents instructions to the user using formatted text and graphics specified using the Hypertext Markup Language (HTML). Instructions can contain hyperlinks to web pages that present additional information on demand in a web browser using text, graphics, and other media.
The Task Guide Editor is shown below. The overview pane (left) contains tabbed windows that display the procedure’s steps and groups, along with the variables and Java library functions that can be used in procedure steps. The details pane (right) lets authors edit the step or group that has been selected in the left pane. The bottom pane lists text search results and problems detected automatically by the Editor.
Figure 13. Task Guide Procedure Editor.
Each step can contain static or dynamic instructions and verifications. Static instructions present the same information each time the procedure specification is executed. Authors use the built-in graphical HTML editor to specify the content and format of static instructions as formatted text, images, and hyperlinks.
You can also specify dynamic instructions by embedding expressions within the instruction’s HTML text. During execution, Task Guide evaluates each embedded expression and replacing the expression with its value. Expressions often contain references to variables whose values can be entered by the user, received from external systems and databases during
B-6
procedure execution, or computed from mathematical, Boolean, or string expressions that refer to other variables. You can also use conditional inclusion to include some text and exclude other text when displaying a step, based on the values of certain procedure variables. Compared to static instructions, dynamically-generated instructions can filter information to present instructions that are more succinct and targeted to the user or situation. They can also select or compute hints, feedback, default values, or recommendations.
Task Guide supports gradual automation, so manual instructions read and followed by users can be replaced over time with steps that use calculations to retrieve data, compute values, and carry out actions automatically. For example, a step could simply provide instructions that tell the user how to carry out a particular task. This step could be enhanced with calculations that automate the retrieval of relevant data. Further enhancement might use calculations to compute default parameter values or decisions and prompt the user to confirm or override them. As confidence increases in the reliability and robustness of the automated recommendations, the organization could replace the step with a fully automated step that analyzes data, decides and acts without user intervention. In this manner, a manual procedure can evolve over time into a more automated one.
Task Guide’s extensible architecture enables integration with general purpose and application-specific Java software libraries that provide functions that are invoked by calculations. This architecture enables Task Guide procedures to incorporate sophisticated automated data retrieval, interpretation, automated reasoning and decision-making algorithms. The Task Guide application programming interface (API) enables you to embed Task Guide within a larger Java application or applet.
C-1
Appendix C
Authoring Tool Usability Questionnaire and Responses
C-2
This appendix presents responses to questions on the usability of the tutorial authoring tool provided by a former instructor at Joint Forces Staff College.
1. What kind of changes/additions do you think an instructor is likely to want make to the
Stability Operations Tutorials?
• Correct/amplify/update readings • Alter/correct/expand questions and/or answers • Add/change display elements to reflect newer images/doctrine or enhanced detail • Alter reference list
For each of these four areas, the primary reason for executing the changes would be to keep current with evolving doctrine and/or policy statements – the recent natural disasters in Haiti and Chile are recent evidence, especially when one considers the manner of response, that stability operations are a constantly changing/evolving field of endeavor. A secondary reason for executing changes will likely be to adapt to either the specific course instructors style or to the variety of students at JFSC, which includes foreign officer fellows utilizing English as a second language. 2. Does the Task Guide authoring tool support the ability to make those changes? Yes, the authoring tool supports making all of the above listed changes, as well as the ability to create a new tutorial from scratch – given the instructional design prepared by the applicable instructor. 3. Please describe your assessment of the usefulness of the capabilities provided by the Task Guide authoring tool.
Given the ability to substantially alter an existing tutorial or create a tutorial from scratch, I find the authoring tool has substantial usefulness. Among the capabilities demonstrated during the training period, I find the following to be most promising:
• Ability to display a wide variety of visual elements in either tutorial or exam sections. • Ability to target either instruction path or exam paths based on student answers. Evaluation
of student answers by the program could include key-word scoring – opening up the possibility that follow-on interactions would be able to fill voids/amplify weak areas vice cover the entirety of a subject. When used as pre-instruction/preparation for a lesson, this would allow for a more efficient and ultimately effective process by best targeting individual student needs.
4. Please describe your assessment of the level of effort needed to learn the tool well enough to make simple changes to the Tutorials. Please describe what you assume to be a simple change.
C-3
Roughly one hour of review of the supporting documentation and 2 ½ hours of on-line collaboration with the developer were necessary to prepare to make the changes described in the answer to question one. The changes listed in the answer to question one are essentially simple changes – editing/updating. In my opinion, the level of training necessary to accomplish these simple changes will by necessity include sufficient training to accomplish more complex changes – altering question structures, adding additional instructional paths, etc. This results from the need to be able to execute changes while understanding the overall structure of the tutorial – necessary to avoid inadvertently altering the structure. 5. Please describe your assessment of the level of effort needed to learn the tool well enough to make more complex changes to the Tutorials. Please describe what you assume to be a complex change.
I would estimate that an additional one to two hours of on-line collaboration preceded by an additional hour of study of the operational guides would be necessary. The bulk of the collaboration time would be spent in monitored/guided conduct of more complex changes. The potential intricacy of connections should in my opinion be laid out directly in front of the prospective user, ideally through at least two to three examples. By complex changes, I have in mind the creation of a tutorial from scratch, the addition of alternate questioning/instructional paths than those currently existing, or the addition of hints and retries to existing questions (not an all-encompassing list, but illustrative). In all cases, it would be beneficial to reinforce the essential connection between an instructional design and the specific technical tasks being completed – the power of the tool will be realized best if the effort spent at instructional design exceeds that of the time required to make the technical changes. 6. Please describe the knowledge, skills, and/or attitudes required by instructors to use the authoring tool successfully.
Attention to detail, advanced preparation of changes (to include flow-charting of more complex changes), detailed knowledge of the subject matter and instructional plan, documentation of changes and version control, basic level of computer familiarity common among instructors who utilize education technology systems/networks (the tool did not appear to require an in-depth knowledge of graphic design tools, C++, or other programming languages). 7. Please describe strengths of the Task Guide authoring tool, especially those that you consider to be relatively unique.
Graphic interactions that appeared to be easily comprehended, useful and well indexed supporting texts, the ability to test changes to verify operability without having to run the full tutorial, the ability to pursue changes at the level of detail required. 8. Please describe changes you would recommend making to the Task Guide authoring tool.
When pursuing changes to some of the question structures, the display included the base operating code language, which could be off-putting to some less programming-savvy users. This is not a large handicap, but interfaces that show as little code as possible and/or simplify the amount of manipulation necessary by the basic user would be beneficial.
C-4
9. Can you identify additional training applications for which the Task Guide authoring tool might be an appropriate tool, especially those that would be useful to your organization? If so, please describe.
One potential application would be to conduct a combination pre-test and remediation process for incoming students. Arriving students can have very different levels of experience and familiarity with the concepts of joint military operations and planning – some will have used them extensively and recently, some will have been instructed in them at a point several years in the past and had limited opportunities to either practice or refresh their knowledge. The challenge is to both determine this level in advance of attendance and to provide the students with useful feedback and recommendations on improving their familiarity. I envision an on-line pretest that would support multiple tracks based on the individual student’s scores and familiarity with “set-up” questions, and then guide them along paths that address weaknesses while providing opportunities to either skim or bypass areas of strength. For the college, these results would be compiled and factor into the assignment of students to seminars to produce a balanced level of expertise and an improved starting level for the course of instruction.
- ACKNOWLEDGEMENTS
- INTRODUCTION
- Domain Analysis
- Design Principles
- Development Process
- Participants
- Materials
- Procedure
- Authoring Tool Development and Usability Assessment
- Tutorials
- Tutorial version 1 – October 2008
- Security in Sudan.
- Governance and justice in Haiti.
- Tutorial version 2 – May 2009
- Introduction to operations in JIIM environments.
- Cross-cultural competence inventory (3CI).
- Tutorial version 3 – November 2009
- The interagency management system in action!
- Tutorial interactions
- Knowledge checks
- Situation judgment tests
- FINDINGS AND DISCUSSION
- Training Effectiveness Evaluation
- Pre- and post-test results
- Reaction survey results
- Limitations of the findings
- Authoring tool usability assessment
- CONCLUSIONS AND RECOMMENDATIONS
- Transition to Operational Use
- Recommendations
- APPENDIX A
- IMS TUTORIAL KNOWLEDGE TEST ITEMS AND SIGNIFICANCE VALUES FOR DISCRIMINATING PRE- AND POST-TEST SCORES
- APPENDIX B
- TASK GUIDE AUTHORING TOOL OVERVIEW
- Task Guide Training Tutorials
- Task Guide Software Suite
- Task Guide Procedures
- Task Guide Authoring Tool
- Appendix C
- Authoring Tool Usability Questionnaire and Responses
