Introduction The Joint Force Protection
The Joint Force Protection Advanced Security System (JFPASS) has been created to
solve the prominent problem in today’s military of protecting troops in a joint environment.
There currently is not a comprehensive method to determine both the quality of architectural
products and of the instantiated system. The Value-Driven Enterprise Architecture (VDEA)
evaluation tool was created to fill this critical requirement.
1.1 General Background
Force protection has taken a prominent role in today’s environment, following the attacks
on the World Trade Center and Pentagon, the United States military has been deployed in a new
kind of warfare. Given the unprecedented warfare tactics (irregular warfare) being employed by
the enemy, protecting personnel and assets is just as important now as it has ever been. To
combat the threats facing the U.S. military, a new emphasis has been placed on joint operations
in which joint warfighting are essential to the current military culture. Therefore, the U.S.
military is seeking to improve the trust and confidence between the separate services and better
employ their individual core competencies to accomplish the mission of the United States more
effectively (Office of the Chairman of the Joint Chiefs of Staff (CJCS), 2007; Joint Chiefs of
Staff, 2004).
The new joint environment has created a need for guidance to govern the combined
operations of the separate services (Office of the CJCS, 2007). There is only general guidance
that dictates the scope and range of each separate service’s responsibilities and individual service
documents that dictate their specific Concepts of Operations (CONOPS). However, the
combined library of guidance documents lacks overarching rules for how joint operations will be
conducted and how the individual services will proceed in an environment where all operations
are handled by a mix of service capabilities.
The systems engineering field has created several tools to represent complex systems,
such as force protection systems. An important tool within the Department of Defense is system
architecture. System architecture allows the user to represent an extremely complex system
through a series of “views” which present the system through a number of perspectives. These
architectures are used in the acquisition of a system and through its life-cycle to document its
development. Judging the quality of the architecture and the systems that it represents, however,
is a challenge. Several models have been created to evaluate different aspects of architecture, but
few focus on the entire portfolio with the instantiated system in mind. These evaluation tools are
generally based on the existing system, as opposed to the needs of the decision-maker. This
effort combines the Operations Research field, with its Decision Analysis tools, with the
Enterprise Architecture field and its Architecture Evaluation tools. Specifically, Value-Focused
Thinking is used to evaluate Systems Architecture at the intersection of these ideas. Figure 1.1
shows a VENN Diagram of the research area.
Figure 1.1. VDEA Venn Diagram
Value-Focused Thinking is an objective decision analysis approach intended to overcome
the problem of multicriteria-decision making. It serves to eliminate assumptions and reveal the
overarching values at the base of a particular decision. Using a set methodology such as this
allows decision-makers to ensure that they are getting the end product that they require and are
expecting to receive. Through an established process of identifying objectives; developing
values, measures, and weights; and then applying functions to these values; a detailed numerical
analysis can be performed to compare alternatives or to evaluate a single alternative and show
areas lacking in the important values (Chambal, 2001).
1.2 Specific Background
To address growing problems of multi-service coordination within the joint community, a
system was proposed which would integrate force protection. The JFPASS project began when a
Joint Capability Technology Demonstration (JCTD) was performed for the Joint Force
Operations
Research
Enterprise
Architecture
Decision
Analysis
Value-
Focused
Thinking
Architecture
Evaluation
Tools
VDEA Scorecard
Protection Advanced Security System (JFPASS) program (Rains, 2008). The JCTD is intended
to demonstrate the integration of various components via a combined command and control
architecture. This architecture will encompass the entire range of Joint Force Protection
functions. It is based upon the joint operational concepts of Detect, Assess, Warn, Defend and
Recover (DAWDR) (IUBIP, 2006).
The goal of the JFPASS effort is to develop an architecture that will represent the
JFPASS system and its Joint Capability Technology Demonstration (JCTD) as required by JCIDS
(Chairman of the Joint Chiefs of Staff (CJCS), 2007). This research effort intends to create an
evaluation tool to evaluate the enterprise architecture, based on stakeholder values. All aspects
of this project are based on the direction given by the IUBIP and centered on the
DAWDR construct. Specifically for this effort, the Detect, Assess, and Warn aspects of DAWDR
are being investigated (Rains, 2008).
1.3 Research Problem
There is currently no method to evaluate the effectiveness of a force protection system
based solely on architectural products. With the Air Force-wide focus on using architecture as a
documentation method and a procurement tracking system, an evaluation method is required for
DoD-specific architectures and specifically in this case for a force protection system architecture.
1.4 Research Objective and Questions
This thesis will determine specific Force Protection values and evaluate an existing
enterprise architecture based on these values. The JFPASS architecture will be evaluated and an
analysis returned including critical deficiencies and required improvements. This research will
determine an appropriate evaluation method for existing architecture and a way to recommend
future courses of action based on a set of existing products.
The questions this research effort will answer are: (1) How can VFT be applied to an
evaluation of a set of architectural products? (2) What is the resulting value hierarchy to
evaluate a force protection system? (3) What are the related weights and measures for the
hierarchy? (4) How well does the provided architecture score based on this hierarchy and where
are the shortfalls and potential areas of improvement?
1.5 Methodology
Value-Focused Thinking will be applied to architecture to evaluate its viability. To date,
VFT has not been used to evaluate an architectural product. In fact, there is very little research
on the topic of evaluating architecture and little to no peer-reviewed research regarding the
analysis of a force protection system. Leveraging VFT, a methodology for architecture
evaluation will be developed.
1.6 Scope
The scope of this thesis will be limited to the architectural products and the environment
within which these products were intended to function. An extendable and defensible tool will
be created to evaluate a set of static architectural products. The scope of the force protection
environment includes worldwide military installations. It is limited, however, to the realm of
joint operations. Therefore, battlespaces controlled by an individual service will not be
addressed. For example, portside security will be addressed, but force protection at sea is not
considered as this is a Navy-specific battlespace. Space assets will also not be included. Threats
from the air will be taken into consideration, but the airspace operating environment will not be
included. Since the Air Force maintains primary control over air space engagements (although
all services operate within this environment), the protection of air assets is not a joint operation.
Within the force protection area, the Detect, Assess, Warn, Defend Recover (DAWDR) construct will
be included in the evaluation, although the Defend and Recover tenets are not of primary concern.
1.7 Review of Chapters/Research Approach
Chapter 2 consists of a review of the available force protection/facility protection
material, as well as the DoD guidance governing the individual service’s force protection efforts.
It discusses how these documents relate to the research effort. Chapter 3 details the methodology
used for this effort. Specifically, it discusses the 10-step VFT process employed here and how
each step was used to create the resulting hierarchy, assign weights, create Single Dimension
Value Functions (SDVF), and analyze the model. It also discusses the collection of relevant
materials and communication with the decision making entity. Chapter 4 provides an actual
evaluation of the architecture in question. It will show how the instantiated system architecture
scored on the hierarchy and areas of improvement to produce a fully functional and effective
force protection system. Chapter 5 discusses these findings and their applicability to the force
protection mission. Chapter 5 also highlights the impact of this effort and details the future
research required to continue this effort, as well as how it can be applied to other areas.
Chapter 2. Literature Review
This chapter presents important previous research pertinent to this effort. Joint force
protection is presented to provide the context for the overall Joint Force Protection Advanced
Security System (JFPASS) project. The field of systems engineering is discussed, as system
architecture is the tool used to produce the product being examined. Decision analysis and
Value-Focused Thinking were used to evaluate the provided architecture. The basis of the value
generation step within Value-Focused Thinking was the affinity diagramming method, which is
taken from the management and planning toolbox. Finally, net-centricity will be summarized as
it applies to this project and its impact on the Department of Defense (DoD).
2.1 Joint Force Protection
The term Joint Force Protection (JFP) is used by the DoD to describe efforts related to
protecting personnel, assets, and information among all service components. Currently, each
service has its own tactics, techniques, and procedures (TTPs) for accomplishing this goal.
Recent developments on the political world stage have caused the DoD to move toward a more
joint environment, as opposed to the separate TTPs formerly used by the services. This idea is
outlined in the National Military Strategy, which states that “achieving the objectives of protect,
prevent, prevail requires connected joint operating concepts (JOCs)” (Joint Chiefs of Staff,
2004). Furthermore, Joint Document 1-02, the Department of Defense Dictionary of Military
and Associated Terms, defines force protection as,
Preventive measures taken to mitigate hostile actions against Department of
Defense personnel (to include family members), resources, facilities, and critical
information. Force protection does not include actions to defeat the enemy or
protect against accidents, weather, or disease. (Joint Chiefs of Staff, 2008, p. 214)
The Protection Joint Functional Concept (PJFC) (Department of Defense,
2004) goes on to state that the actions involved in force protection (FP) are intended
to conserve the force’s fighting potential so that it may be applied at the appropriate
time to accomplish the mission at hand (Department of Defense, 2004). The U.S.
definition also aligns very closely with the North
Atlantic Treaty Organization (NATO) definition of force protection (NATO Standardization
Agency, 2008). This connection allows better communication among multi-national forces.
The military’s current method of assessing FP in a facility is through the use of Joint Staff
Integrated Vulnerability Assessment (JSIVA) teams. These teams of force protection experts
visit military installations to determine their level of protection. As part of this assessment, they
provide the required training and feedback to enhance protection postures at these installations.
The JSIVA program provides a comprehensive assessment tool for operational facilities, but they
have no method for evaluating FP systems under design (Cirafici, 2002).
2.1.1 National Policy
Joint force protection concepts are drawn from the national strategic objective. National
guidance regarding force protection and military operations come in several tiers. The National
Security Strategy (NSS) is the Presidential directive which guides all efforts to secure and defend
the United States. It discusses international strategy as well as the United States’ goal of
improving the quality of life not only within the U.S., but in other countries as well. It also
discusses the strategic objective of eliminating terrorism by winning the War on Terrorism
(Office of the President of the United States of America, 2002).
The National Defense Strategy (NDS) directly supports the NSS by establishing
objectives by which the goals of the NSS will be accomplished and measured. The NDS
provides the link between the DoD and other government agencies as they relate to the security
objectives of the nation. The objectives set forth by the NDS are to: (1) secure the United States
from direct attack, (2) secure strategic access and retain global freedom of action, (3) establish
security conditions conductive to a favorable international order, and (4) strengthen alliances and
partnerships to contend with common challenges (Office of the Secretary of Defense, 2008).
These objectives provide the direction for the National Military Strategy (Office of the Secretary
of Defense, 2008).
The National Military Strategy (NMS) provides the focus for military activities by
specifying the overall objectives set forth in the NSS and NDS. To this end, the NMS refers
specifically to three guiding ideas. “Protect the United States” refers specifically to what has
become known as “Homeland Security.” The NMS establishes homeland security as the first
priority of the United States. The armed forces are responsible for securing the nation, both at
home and abroad. The military accomplishes missions outside the U.S. to counter threats as they
occur at their source. They must then secure strategic approaches to the U.S. to ensure enemy
forces cannot gain direct access to the country. Lastly, they must employ force as directed on
home soil in the case of direct attack. “Prevent conflict and surprise attack” is the second idea
specified in the NMS. This refers mainly to strengthening alliances and creating a security
environment in which aggressions from adversaries is discouraged. Preventing this conflict is a
goal which requires global action and attention to any adversary who may pose a threat to the
United States. “Prevail against adversaries” is the objective that refers specifically to the
military’s mission of swiftly defeating adversaries in campaigns and wars. This objective
includes the ability to integrate all available technologies, capabilities, and information in
overlapping campaigns (Joint Chiefs of Staff, 2004).
Accomplishing the objectives of protect, prevent, and prevail requires the use of Joint
Operational Concepts (JOCs). The NMS focuses largely on the concept of a Joint Service. The
desired attributes of a joint force are one that is: fully integrated; expeditionary; networked,
decentralized, and adaptable; has decision superiority; and is capable of lethality. The scope of
security for the joint force is defined in the NMS as:
the ability to operate across the air, land, sea, space and cyberspace domains of
the battlespace. Armed Forces must employ military capabilities to ensure
access to these domains to protect the Nation, forces in the field and U.S. global
interests. The non-linear nature of the current security environment requires
multi-layered active and passive measures to counter numerous diverse
conventional and asymmetric threats. These include conventional weapons,
ballistic and cruise missiles and WMD/E. They also include threats in
cyberspace aimed at networks and data critical to U.S. information-enabled
systems. Such threats require a comprehensive concept of deterrence
encompassing traditional adversaries, terrorist networks and rogue states able to
employ any range of capabilities. (Joint Chiefs of Staff, 2004, p. 18)
By defining joint force security, the National Military Strategy provides the context for joint
force protection. It establishes a focus on joint operations and on directing the actions of the
military. The NMS implies a need to protect those who are accomplishing the mission. This
implication is explored in more depth in the implementation of the objectives set forth in the
NMS by joint guidance documents.
2.1.2 Joint Guidance Documents
In addition to national policy, several joint documents have been created to help guide the
development of the joint force. Each of these documents is targeted toward a specific audience
for a specific purpose. There is overlap to each of them, but their guidance is standard across the
documents. The recurring theme is that the service components must learn to operate effectively
in a joint environment. Each of the following documents gives information critical to operating
jointly.
2.1.2.1 Joint Publication 1
Joint Publication 1 (JP1) is the overarching guidance for all other joint publications. It
provides the guidance for the unified operations of all branches of service by bridging policy and
doctrine. This common perspective is employed by all commanders to ensure that each service
component is working toward the same goal. This document directs the services to operate
jointly by relying on each other’s skills and capabilities. JP1 states that despite the U.S.
military’s ability to conduct warfare, the military must also focus on the strategic security
environment to ensure the viability of its warfighting capability (Office of the CJCS, 2007).
2.1.2.2 Joint Publication 3-0
Joint Publication 3-0 (JP3) extends the guidance in JP1 to include planning and execution
across the range of military operations typically found in the joint environment. In JP3,
protection is included as a critical joint function. Four primary protection functions are outlined
as active defensive measures, passive defensive measures, applying technology and procedures,
and emergency management and response (Office of the CJCS, 2008). JP3 also extends force
protection to include friendly nations and other allied organizations. It also discusses health
protection as a subsection of FP.
JP3 states that the protection function itself includes several tasks. Each task directly
relates to the Detect, Assess, Warn, Defend, Recover (DAWDR) construct and concept of
protecting personnel, assets, and information. Air, space, and missile defense; protection of
noncombatants; physical security; antiterrorism; and eight other tasks comprise the protection
function (Office of the CJCS, 2008). These protection tasks show the full range of force
protection.
2.1.2.3 Protection Joint Functional Concept
The Protection Joint Functional Concept (PJFC) is a DoD governing document regarding
protection of friendly personnel, information, and assets. It is intended to guide future joint
operations within all service components. The PJFC defines protection as “the ability to sense
adversary activities, understand their impact on Joint Force operations, and make timely and
appropriate decision to execute capabilities to neutralize or mitigate adversary effects”
(Department of Defense, 2004). This document identifies the three key areas for protection:
Personnel, Assets, and Information. They are defined by the protection construct shown in
Figure 2.1.
C F O
Monitor Understand Decide Execute
C F P
Detect Assess (Decide & TaskWarn )(Active & PassiveDefend )Recover
M C A Protect
Personnel Protect Physical Assets Protect Information
M C E
•Air and Missile Defense •Maritime Defense
•IED Defense •Defensive Counter-Space
•Combating WMD •Non Combatant Evacuation
•Force Health Protection •Others…
Figure 2.1. The Protection Construct (Department of Defense, 2004)
This construct defines the five key aspects of force protection: Detect, Assess, Warn,
Defend, and Recover (DAWDR). The joint force commander must be able to effectively execute each
of the DAWDR tenets in a joint environment. The PJFC also provides the context and overall guidance
for each of the other Joint Functional Concepts, such as Battlespace Awareness, Command and Control,
and Force Application. It also addresses the Mission Capability Areas (MCAs) and Mission Capability
Elements (MCEs), which are the specific protection tasks which enable the joint force to execute its
mission (Department of Defense, 2004). The hierarchy described in Figure 2.1 provides the context
for joint force protection. Specifically, it defines the scope of joint force protection as falling within
the DAWDR construct. It then specifies DAWDR to include personnel, assets, and information,
providing a more specific scope for the objective of force protection.
2.1.2.4. Universal Joint Task List (UJTL)
The Universal Joint Task List (UJTL) provides the standardization required to plan,
conduct, evaluate, and assess joint and multinational training. It dictates what specific functions
must be accomplished by the joint force. When combined with the Service Task Lists, it
provides a comprehensive list of tasks and measures for all levels of the DoD. The UJTL also
provides the context for interoperability; however, it does not define how services are expected
to interact with each other in the execution of the joint mission. The tasks are divided into
Strategic National, Strategic Theater, Operational, and Tactical tasks. Each of these tasks
includes lists of subtasks that fall under the major idea. Under tactical tasks, subsection six
focuses specifically on force protection. The operational context for each task is defined by the
joint conditions section. Joint Condition 2.7 is the section which focuses on the protection of
each area of air, sea, and land. The most critical part of the UJTL to this study is the definition of
the measures associated with each protection area. For example, the measures of Air Superiority
are defined as Full, General, Local, or No. The UTJL, however, only provides broad guidance
and does not dictate how each service will fulfill its mission (Joint Staff, 2002).
2.1.3 Service Policies
Under the joint guidance, each individual service component must create its own force
protection guidance to dictate how the principles set forth in joint and national doctrine will be
accomplished. Military installations are controlled by the owning service component; however,
in the case of joint bases or shared installations, a single service is chosen as the lead for force
protection on that installation. This presents problems because of the different implementations
of the joint guidance. Each service operates within the guidelines set forth, but executes those
guidelines differently.
2.1.3.1 Air Force
The Air Force’s Installation Security Program (ISP) is their primary guidance for force
protection. Air Force Instruction (AFI) 31-101 provides the implementation of Air Force Policy
Directive 31-1, Physical Security. The Air Force places primary responsibility for force
protection within their Security Forces career field. This single career field is responsible for
creating programs and regulations to ensure that the entire population of the installation is secure.
Security Forces are responsible not only for the implementation of the Air Force ISP, but they are
also responsible for ensuring that the installation complies with each level of guidance. They
must create and maintain an ISP, as well as host the Installation Security Council (ISC) (HQ
AFSFC/SFON & SFOP, 2003). A portion of the Air Force force protection responsibility falls to
the Civil Engineer career field, as they are responsible for designing and building both home
station and expeditionary structures which must comply with the Anti-Terrorism/Force
Protection (ATFP) guidelines as well as the doctrine set forth in joint publications and
installation-specific regulations.
2.1.4.2 Army
The Army’s physical security program is a component of its force protection program.
The Army’s program relies on the military police force, but it also makes use of all other soldiers
to implement the policies and guidance. For example, physical security inspectors can be from
any military Occupational Specialty (Department of the Army, 1993). This policy makes force
protection a more implicit responsibility. The regulations within the Army are carried out by
programs put in place at higher headquarters, but compliance is a command responsibility.
2.1.3.3 Navy/Marine Corps
The Navy and Marine Corps have an entirely different approach to force protection.
Since they spend the majority of their time at sea, there is a command within the Navy known as
the Force Protection Command. Its primary duty is to protect Naval forces from Naval threats.
The Navy’s port security program is managed either by civilians or their ship security personnel
(NTTP 3-07.2.1, 2003; NWP 3-07.2 (Rev A), 2004).
2.1.4 Integrated Unit Base Installation Protection (IUBIP)
Integrated Unit Base Installation Protection (IUBIP) has three guiding documents: the
IUBIP Concept of Operations (CONOPs), the IUBIP Functional Area Analysis (FAA), and the
IUBIP Joint Capability Document (JCD). The IUBIP CONOPs “conceptualizes the integration
of protection capabilities for agile, decisive, and integrated force employment in all phases of
combat and supporting operations” (IUBIP, 2006). The IUBIP FAA defines the tasks required of
the joint force to accomplish the goal of protecting personnel, information, and assets (Joint
Chiefs of Staff, 2007). The IUBIP JCD discusses the Joint Functional Areas, as well as the
required capabilities, capability gaps, and threat environment (Joint Requirements Oversight
Council, 2007).
These documents provide the context for the specific project. The IUBIP CONOPs,
which builds upon the FAA and JCD, defines the military problem for which the Joint Force
Protection Advanced Security System (JFPASS) is being created. Specifically, as the joint force
is reduced due to budgetary constraints, it will require more efficiency to continue its current
level of operations. The IUBIP states that the joint force has a great deal to gain from integration
and explains how this may be accomplished. The CONOPs discuss the benefit that net-centricity
will have on a newer, integrated joint force and how net-centricity is required as the joint force
matures (IUBIP, 2006). Net-centricity is typically defined as the operation of a group of nodes in
communication with each other.
2.2 Systems Architecture
With the complexity of force protection, a system of analysis is required to gain an
understanding of how the individual services interrelate. The systems engineering field is an
interdisciplinary approach to enable the realization of successful systems (Blanchard &
Fabrycky, 2006). The creation and design of these systems is accomplished through graphical
representations, design tools, and process analysis. Through the use of these tools and system
analysis, system designers and managers can better understand and therefore manage their
systems.
One of the tools within the systems engineering field that has gained wide use and
acceptance within the DoD is systems architecture. The DoD Architecture Framework (DoDAF)
was written as the authoritative source on DoD’s use and implementation of architecture. The
DoDAF describes architecture as “the structure of components, their relationships, and the
principles and guidelines governing their design and evolution over time” (DoD Architecture
Framework Working Group, 2007). Maier and Rechtin (2002) refer to architecture simply as
“the art and science of designing and building systems.” The DoDAF prescribes a systematic
process of architecture by providing the standards by which architecture “views” or products
(discussed in Section 2.2.1) are created, while Maier and Rechtin (2002) tend to believe that
architecture is a more abstract concept which requires a “process of insights, vision, intuitions,
judgment calls, and even taste.”
2.2.1 DoDAF
The DoDAF is the result of at least 12 years of evolution of DoD policy and procedures
(DoD Architecture Framework Working Group, 2007). It began with the Command, Control,
Communications, Computers, Intelligence, Surveillance and Reconnaissance (C4ISR)
Architecture Framework v1.0 in 1996 followed by v2.0 in 1997. The DoDAF v1.0 was
subsequently released in 2003. The current version of DoDAF is 1.5 and was released in 2007.
DoDAF v2.0 has been released in draft form and is being coordinated for an official release
expected by the middle of 2009. All of these efforts are the result of the move toward joint and
multinational operations (DoD Architecture Framework Working Group, 2007). The DoD has
ensured the use of DoDAF through the use of policies and directives which require its use in
acquisition processes. Table 2.1 displays the evolution of the policy documents that have
directed the use of DoDAF.
Table 2.1. Federal Policy for Architectures (DoD Architecture Framework Working
Group, 2007)
Policy/Guidance
Description
Clinger-Cohen Act of 1996
Recognizes the need for Federal Agencies to improve the
way they select and manage IT resources and states
information technology architecture, with respect to an
executive agency, means an integrated framework for
evolving or maintaining existing IT and acquiring new IT to
achieve the agency’s strategic goals and information
resources management goals”. Chief Information Officers are
assigned the responsibility for “developing, maintaining, and
facilitating the implementation of a sound and integrated
IT architecture for the executive agency.”
Office of Management and Budget
Circular A-130
“Establishes policy for the management of Federal
information resources” and calls for the use of Enterprise
Architectures to support capital planning and investment
control processes. Includes implementation principles and
guidelines for creating and maintaining Enterprise
Architectures.
E-Government Act of 2002
Calls for the development of Enterprise Architecture to aid in
enhancing the management and promotion of electronic
government services and processes.
OMB Federal Enterprise
Architecture Reference Models
(FEA RM)
Facilitates cross-agency analysis and the identification of
duplicative investments, gaps, and opportunities for
collaboration within and across Federal Agencies. Alignment
with the reference models ensures that important elements of
the FEA are described in a common and consistent way. The
DoD Enterprise Architecture Reference Models are aligned
with the FEA RM.
OMB Enterprise Architecture
Assessment Framework
(EAAF)
Serves as the basis for enterprise architecture maturity
assessments. Compliance with the EAAF ensures that
enterprise architectures are advanced and appropriately
developed to improve the performance of information
resource management and IT investment decision making.
General Accounting Office
Enterprise Architecture
Management Maturity Framework
(EAMMF)
“Outlines the steps toward achieving a stable and mature
process for managing the development, maintenance, and
implementation of enterprise architecture.” Using the
EAMMF allows managers to determine what steps are
needed for improving architecture management.
There are several types of architecture in systems engineering resources. DoDAF
discusses Integrated Architectures, Composite Architectures, and Federated Architectures.
Integrated architectures are those in which there is concordance between all products and
entities. They use a standard nomenclature throughout the operational views (OV), systems and
services views (SV), all views (AV), and technical views (TV). In this case, operational views
are those which describe the general tasks, activities, and major information exchanges. Systems
and services views capture specific interconnection information further specifying the
information found in OVs. Technical views contain the minimum set of rules which govern the
functions of the system or system elements. All-views are the overarching informational views.
They provide information about the architecture, but do not actually show an architectural view.
Table 2.2 shows all views included within DoDAF. Integrated architectures facilitate ease of use
and communication, as well as aggregation of information. Composite architectures are those
composed of separate parts. Generally, several integrated architectures are pulled together to
form composite architectures which support a more broad set of goals. Finally, federated
architectures are distributed information bases compiling information of use to decision-makers
at higher levels. All architectures increase the net-centricity of a system by encouraging the
process of examining links between nodes and modeling the composition of the system.
The DoDAF’s use has become commonplace within several areas of the DoD, particularly within
the acquisition process. The Joint Capabilities Integration and Development System (JCIDS)
formally defines the acquisition process and directs the use of certain architecture products for
milestone decision points (CJCS, 2007). For example, Milestone Decision Point A requires an
OV-1 view for consideration of the project (CJCS, 2007).
Table 2.2 DoDAF Views (DoD Architecture Framework Working Group, 2007)
Applicable View
Framework
Product
Framework Product Name
All View
AV-1
Overview and Summary Information
All View
AV-2
Integrated Dictionary
Operational
OV-1
High-Level Operational Concept Graphic
Operational
OV-2
Operational Node Connectivity Description
Operational
OV-3
Operational Information Exchange Matrix
Operational
OV-4
Organizational Relationships Chart
Operational
OV-5
Operational Activity Model
Operational
OV-6a
Operational Rules Model
Operational
OV-6b
Operational State Transition Description
Operational
OV-6c
Operational Event-Trace Description
Operational
OV-7
Logical Data Model
Systems and Services
SV-1
Systems/Services Interface Description
Systems and Services
SV-2
Systems/Services Communications Description
Systems and Services
SV-3
Systems/Services-Systems/Services Matrix
Systems and Services
SV-4a
Systems Functionality Description
Systems and Services
SV-4b
Services Functionality Description
Systems and Services
SV-5a
Operational Activity to System Function Traceability Matrix
Systems and Services
SV-5b
Operational Activity to System Traceability Matrix
Systems and Services
SV-5c
Operational Activity to Services Traceability Matrix
Systems and Services
SV-6
Systems/Services Data Exchange Matrix
Systems and Services
SV-7
Systems/Services Performance Parameters Matrix
Systems and Services
SV-8
Systems/Services Evolution Description
Systems and Services
SV-9
Systems/Services Technology Forecast
Systems and Services
SV-10a
Systems/Services Rules Model
Systems and Services
SV-10b
Systems/Services State Transition Description
Systems and Services
SV-10c
Systems/Services Event-Trace Description
Systems and Services
SV-11
Physical Schema
Technical Standards
TV-1
Technical Standards Profile
Technical Standards
TV-2
Technical Standards Forecast
2.2.2 Architecture Evaluation
Architecture evaluation has taken many forms from quantitative scoring measures to
simple heuristics. It is of great value to not only the model builder, but to the project sponsor as
well, to be able to determine the quality of a set of architectural products and the associated
instantiated system. Since architectures are intended to represent a system, it is important to rate
not only the architectural products themselves, but also how they accomplish the goal of
representing the needs of the system itself.
One such evaluation method is Ford’s i-Score (Ford, Graham, Colombi, & Jacques,
2008), which measures the interoperability of the architecture. To do so, it uses the DoDAF
OV5, OV-2, and SV-3. It is similar to Value-Focused Thinking (VFT) in that it yields a single
quantitative score that represents how the architecture is performing in terms of interoperability
(Ford, Graham, Colombi, & Jacques, 2008). Levis, Shin, and Bienvenu (2000) discusses the
concept of executable architectures, which relies on modeling and simulation to determine the
effectiveness of a system. There have been other tools as well, such as the Architecture Based
Evaluation Process (Dietrichs, Griffin, Schuettke, & Slocum, 2006) and the Architecture
Tradeoff Analysis MethodSM (Kazman, Klein, & Clements, 2000), which is intended to evaluate
software architectures. Of the architecture evaluation methods in existence, none attempt to
grade an architecture based on “-illities,” nor do they provide a comprehensive generalized
approach in line with the stakeholder’s values.
2.2.3 “Ilities”
“Ilities” have grown in popularity across the quality management field. Particularly in
the areas of software development and systems engineering, they have become standards for
describing system attributes. The INCOSE Systems Engineering Handbook version 3.0 defines
ilities as “the operational and support requirements a program must address (e.g. availability,
maintainability, vulnerability, reliability, supportability, etc.)” (International Council on Systems
Engineering, 2007). Within the systems engineering field, ilities tend to describe the quality
attributes of a system through their descriptive nature. This makes ilities useful for describing
the quality of both the instantiated system as well as the architectural products. Although a
single authoritative source for a list of ilities does not exist, they are often created based on the
quality needs of the system. Several studies and articles refer to individual use of ilities. Ross
(2006) discusses ilities (flexibility, adaptability, scalability, robustness) which describe the
“traditional system design concerns.” McManus et al. (2007) created a quantitative measure for
describing certain system ilities. Their work defines and describes six ilities (robustness,
versatility, changeability, flexibility, scalability, and survivability) which share some of the same
attributes as other studies. These studies begin to provide a framework for the evaluation and
quantification of ilities, although much more work is required in this area. It is possible to
“create” ilities by simply adjusting the tense of a system attribute. A web search yields one list of
63 ilities, with many others scattered throughout various sources.
2.3 Decision Analysis
Evaluating the protection status at a United States military installation is currently a very
subjective process. Each service has inspection methods in place to ensure compliance with
regulations and security procedures, but there is no quantitative, objective method for achieving
this goal. Inspection procedures consist of checklists that are accomplished by subject matter
experts appointed by higher headquarters, but their evaluations are still based on their own
subjective understanding of the regulations. In addition, these evaluations may not match
fundamental joint force protection values.
The Decision Analysis (DA) field provides decision-makers a set of tools for making
decisions that are more objective. In this case, DA provides a quantitative, more objective
approach to accomplishing the goal of force protection evaluations. The methods provided
within the decision analysis discipline give the decision-maker more insight to the problem and
ensure all data is being examined, thereby facilitating better decisions. DA is particularly useful
when several objectives exist and affect different groups of stakeholders. In the case of
architecture, a set of products exist, which serve to document a collection of design decisions.
Decision analysis provides a tool to evaluate the previously made design decisions as they relate
to stakeholder values, as well as a method to evaluate decision opportunities.
Decision analysis provides a systematic, iterative approach (shown in Figure 2.2) to
solving a problem (Clemen & Reilly, 2001). It begins with the most basic step of any analysis, to
identify the decision situation and understand its objectives. In this case, the design of a force
protection system is being evaluated. The system being designed will be subjected to the
evaluations of the DoD acquisition process. The evaluations for system acquisition are largely
subjective and depend on the opinions of the sponsor and acquisition officer.
2.4 Value-Focused Thinking
Value-Focused Thinking (VFT) is a decision making tool developed by Keeney (1992); it
enables decision-makers to look beyond the list of available alternatives and focus on the values
or objectives that are actually important to them in the outcome of the situation. The VFT
method is intended to get decision-makers closer to solutions that they actually want (Keeney,
1992). In a joint force protection system situation, this facilitates the system sponsors getting the
product they need as opposed to choosing between presented alternative systems.
Figure 2.2. Decision-Analysis Process Flowchart (Clemen & Reilly, 2001, p. 6)
2.4.1 Alternative-Focused Thinking versus Value-Focused Thinking
The second step of the process shown in Figure 2.2, “Identify Alternatives,” is the issue
that defines any decision-making problem. A decision problem generally occurs when a
decision-maker is presented with at least two alternatives (Keeney, 1992). They must decide
among the best of the presented alternatives. This approach has been called “AlternativeFocused
Thinking” (AFT) since the decision approach is based on choosing from a finite set of
alternatives. Because AFT happens after a decision problem has been framed and the solution is
chosen from a finite set of alternatives, it is a reactive procedure (Keeney, 1994). VFT, however,
is a proactive approach in which decision-makers examine a problem before the decision is
forced upon them. It enables the decision-makers to determine what is important to them in
advance of making the decision and generate alternatives rather than choosing from existing
alternatives. This approach also ensures that all possible alternatives are considered instead of
only a limited set.
In Alternative-Focused Thinking methodologies, the decision-maker begins the process
with a set of existing alternatives. Alternatives though, are only the means to achieve the values
of the decision-maker in this method. For this reason, the values should be determined before
alternatives are created (Keeney, 1994). The Alternative-Focused Thinking approach leads to
less understanding of what is important in the end goal.
VFT is intended to lead decision-makers to better decisions. The process set forth in
previous studies and literature allows for many other advantages as well. Among these are
uncovering hidden objectives, guiding information collection, improving collection, facilitating
involvement in multiple-stakeholder decision, avoiding conflicting decisions, evaluating
alternatives, creating alternatives, and identifying decision opportunities (Keeney, 1992)
When VFT is used early enough in a process, many alternatives and opportunities for
improvement are presented to the decision-maker. In many cases, a decision situation is forced
and does not leave decision-makers time to evaluate values and create an exhaustive list of
alternatives. Instead, they are presented a finite list of alternatives and must choose the best of
those available. By applying VFT early, it is possible to guide information collection and
identify decision opportunities prior to the decision situation. During the design phase of an
acquisition project, there are several opportunities for improvement over the long process.
Focusing on values throughout the entire process will allow the designer to achieve a more
robust and useful product for the sponsor, instead of simply fulfilling the requirements.
Kirkwood (1997) discusses several of the same advantages, including that the VFT process is
helpful in facilitating communications between stakeholders. VFT seeks to create a hierarchical
representation of what is important to the decision-maker. This hierarchy includes tiers of value,
which become more specific as the tiers progress. The value hierarchy gives stakeholders a
common frame of reference for what is important in the project and gives all stakeholders input
into the importance of each value. In projects with a large number of objects, complex issues,
and many stakeholders, communication is essential to achieving the objectives of the project.
2.4.2 Discussion of Value
Throughout the literature regarding Value-Focused Thinking and Decision Analysis, the
terms “Value” and “Objective” are often used interchangeably. “Values are what we care about”
(Keeney, 1992). They are the fundamental part of any decision that dictates in what the decision-
maker is truly interested. Keeney (1992) uses two distinct terms, value and objective, when
discussing what is important. The term “value” refers to an idea that the decision-maker is trying
to describe. The term “objective” is typically used to describe the evaluation measure of the
value included in the hierarchy. Kirkwood (1997) defines the connection between a value and an
objective by his definition of the term “objective.” He defines it as “the preferred direction of
movement with respect to an evaluation consideration.” Several previous research efforts have
used either term to describe the actual elements of the hierarchy. Shoviak (2001) uses the term
“objective” almost exclusively, while Katzer (2002) uses “value” in the same way, in the same
contexts. An examination of several other VFT-focused research efforts has yielded similar
differences in the use of the terms. Clemen and Reilly’s (2001) discussion of objectives and
values distinguishes the two terms based on their relation to the user. They refer to value as
anything that matters to the decision-maker. Objectives are defined as the specific things that the
decision-maker wants to achieve. Based on these definitions, the combination of the objectives
gives the decision-maker their overall values. The objectives are the specific things that will
influence the final decision and the values are characteristics of the preferred outcome.
2.4.3 Value-Focused Thinking Methodology
VFT was initially laid out by Keeney (1992) and refined by Kirkwood (1997). Over the
years, this methodology has been applied and adapted for several purposes. Keeney (1992)
discussed three “situation based” five-step processes. These processes refer to situations in
which a decision problem or decision opportunity exists. The decision opportunities are then
broken down into two processes. One for a situation before strategic objectives have been
specified and one for situations after strategic objectives have been specified. Each of these
situations have a five step process for completing the VFT process. These processes, in
combination with Kirkwood (1996) can be extended to a ten-step process (Chambal, 2001). The
ten-step process expands on Keeney’s by adding individual steps for measures, value functions,
and hierarchy as well as expanding the analysis of the VFT process by including Kirkwood’s
methods. These methods are shown in Table 2.3.
As demonstrated by the selected methodologies in Table 2.3, there are several different
ways to apply Value-Focused Thinking to a decision. Each of the steps laid out in these
processes share some similar features, which can be combined into a single, ten-step process,
accounting for all major activities and milestones. The ten-step version guides the evaluator
through the Keeney and Kirkwood methodologies in a straightforward fashion, ensuring that
each iterative step accomplishes the necessary activities. The ten-step process effectively
combines the previous methodologies and accounts for all of their ideas. This process has been
refined and applied in several research projects. These projects include an examination of
advanced academic degree profiles (Gentil, 2007), a Force Protection Battlelab project
evaluation initiative (Jurk, 2002), and strategic airlift (Tharaldson, 2006). Chambal’s (2008)
process follows a path of distinct activities, separating Value Hierarchy creation, Measure
Creation, Weighting, Scoring, and Analysis in a unique way. Figure 2.3 shows a graphical
representation of the ten-step process. In the following pages, the VFT process will be discussed
in depth.
Table 2.3. VFT Methodologies (Keeney, 1992; Kirkwood, 1997; Chambal, 2001)
Author
Keeney
Keeney
Keeney
Kirkwood
10-Step Process
Situation
Decision Problems
Decision Opportunities before
specifying objectives
Decision Opportunities
after specifying objectives
All
All
Steps
Recognize a decision
problem
Indentify a decision
opportunity
Specify Values
Identify Decision
Problem Identification
Specify Values
Specify Values
Create a Decision
Opportunity
Structure Objectives
Create Value Hierarchy
Create Alternatives
Create Alternatives
Create Alternatives
Develop Evaluation Measures
Develop Evaluation Measures
Evaluate Alternatives
Evaluate Alternatives
Evaluate Alternatives
Develop Alternatives
Create Value Functions
Select an Alternative
Select an Alternative
Select an Alternative
Determine Single Dimensional
Value Function
Weight Hierarchy
Develop Weights
Alternative Generation
Determine Overall Values for
Alternatives
Alternative Scoring
Select Alternative
Deterministic Analysis
Sensitivity Analysis
Recommendations
Presentation
Figure 2.3. Ten-Step Process Graphical Representation (Chambal, 2001)
Parnell describes three “levels” of VFT models as the Silver, Gold, and Platinum
standards (Parnell, 2007). These standards are used throughout the process as methods of
communicating and building the model. The framework decided upon is descriptive of how the
process is completed. Several aspects of the VFT process are impacted, such as how the
hierarchy is built, description of the problem, construction of measures, and perhaps of most
importance, the development of value weights. The Silver standard is the least preferable of the
standards (Dawley, Marentette, & Long, 2008). It incorporates the opinions of a large number of
stakeholders and uses various idea-generation techniques to determine inductively the values of
the organization (Chambal, 2001). The Gold standard bases the model construction on existing
documents and guidance. Through an examination of documents, such as vision statements,
mission statements, rules, regulations, etc., the model builder is able to gain an appreciation for
what is important to the organization, which allows deductive development of the model. The
organization’s senior leaders must then validate gold standard models (Chambal, 2001). Through
examinations of documentation as well as validation, this standard is most easily defendable
(Jurk, 2002). The Gold standard also allows the model builder to create a “strawman” hierarchy
from which to begin and base discussion. Strawman hierarchies tend to facilitate discussions
with the decision-maker and make effective and efficient use of time (Katzer, 2002). Finally, the
Platinum standard relies on interviews regarding the values of the key decision-maker as well as
technical experts and stakeholders. This method gives not only the Subject Matter Experts
(SMEs) buy-in to the hierarchy, but also facilitates direct involvement of the final decision
authority. The platinum standard often begins with an examination of strategic objectives,
organizational plans, and visions (Chambal, 2001), but moves on to capture the values of the
final decision-maker. This final decision authority’s opinions and views on the system lead to a
more accurate depiction of what is important in the model. These models tend to capture most
accurately the intended hierarchy structure due to the direct involvement of the stakeholders and
final decision authority (Braziel, 2004).
2.4.3.1 Step 1 – Problem Identification
Problem identification is the cornerstone of any scientific process and is consistently
referenced as the first step to solving problems. In many cases, an undesirable solution to a
problem is based on a decision-maker’s failure to identify and understand the problem itself. In
addition to the model builders and decision-maker understanding the objectives of the process,
all stakeholders should have a clear understanding of the goal. Everyone involved in the process
must have a common idea of the problem itself, so that wasted effort can be avoided.
2.4.3.2 Step 2 – Create Value Hierarchy
A value hierarchy is a graphical representation of the values or objectives most essential
to the decision-maker. Keeney (1994) defines values as “Principles for evaluating the
desirability of any possible alternatives or consequences.” The hierarchical structure allows the
model builder to represent values from a top-down perspective, showing not only what the
overarching value is, but also going into the level of detail required for the problem. The
resulting hierarchy must be defendable. A defendable architecture must agree with the
decisionmaker’s objectives as well as the organizational goals. This must also be done within the
constraints of the methodology chosen.
2.4.3.2.1 – Generating Values
The generation of values depends greatly on the standard chosen for the model being
constructed. The actual process of finding these values can be quite different depending on the
choice between the “silver,” “gold,” or “platinum” standard. The values may come directly from
an examination of documentation or from interviews with various personnel and ultimately a
validation by some level of decision-making authority. If possible, the highest-level
decisionmaker should be chosen (Keeney, 1994).
Keeney (1994) distinguishes between two different types of objectives or values. He
refers to fundamental objectives as “[objectives that] concern the ends that decision-makers
value in a specific decision context” and means objectives as “methods to achieve ends.” Means
objectives serve as way to identify fundamental objectives. The means objectives can be
quantified by continually asking the question “Why is that important?” until a fundamental
objective is reached (Keeney, 1994). Fundamental objectives can also be called “ends
objectives” (Kirkwood, 1997).
Developing the list of values can be accomplished through a number of techniques.
These techniques include: develop a wish list; identify alternatives; consider problems and
shortcomings; predict consequences; identify goals, constraints, and guidelines; consider
different perspectives; determine strategic objectives; determine generic objectives; structure
objectives; and quantify objectives (Keeney, 1994). Using Keeney’s (1994) suggested
techniques, one will develop a list of items including fundamental objectives and means
objectives. This list must then be examined to determine what each of the list items are, thereby
eliminating items that are not values. The goal of this process is to end with a list of
“collectively exhaustive and mutually exclusive” objectives (Kirkwood, 1997). These objectives
should be characterized by three features: a decision context, an object, and a direction of
preference.
2.4.3.2.2 – Structuring Values
Once values have been determined, they must be placed into a readable, understandable,
graphical structure. This structure allows for easy communication to a wider range of users.
Keeney (1992) notes that prior to his work, there had not been a standard format for structuring
values. He therefore proposes the hierarchical method of structuring (Keeney, 1992). Kirkwood
(1997) defines a value hierarchy as “a value structure with a hierarchical or “treelike” structure.”
This process is also known as a “top-down” structure, as it is based on the fundamental value.
The basic nature of hierarchies is both vertical and horizontal. As demonstrated in Figure
2.4, the hierarchy is made up of both tiers and branches. Tiers are the layers or levels that,
collectively, specify the values on the tier above. Branches are the values that actually specify
the value above. Within each branch, the values in each progressively lower tier specify a single
value from the above tier of the same branch. The breadth of the hierarchy is defined by the
number of branches and the depth is defined by the number of tiers.
Figure 2.4. Example Hierarchy (Chambal, 2001)
Keeney’s (1992) method involves three basic steps: placing the overall fundamental value
at the top of the hierarchy, relating values on different levels, and stopping the structuring
process. In this process, the overall value is the reason for the decision and defines the breadth
of the decision problem. Choosing this overall value is therefore very important. For some
decisions, it is easy to identify, but ensuring that it is the correct value will affect the entire
process. Following the selection of a fundamental value, other values must be placed below it in
the proper branch and tier based on their relation to the fundamental value (Keeney, 1992;
Kirkwood, 1997). Each progressively lower tier specifies the values above it. In Tier 2 of
Figure 2.5, Values 1 and 2 further define the value found in Value 1 of Tier 1. The combination
of the two lower values makes up the higher value. The critical part of Keeney’s (1992) process
is stopping the structuring process. The “test of importance” is the first determination as to how
many values should be included. The size of the hierarchy must be balanced with the detail
necessary to capture the values. Each value must also be measurable by an attribute (Keeney,
1992). The model builder should continue moving down the hierarchy, progressively refining
the values within each branch by adding more tiers, until the model builder no longer must ask
“What do you mean by that?” (Katzer, 2002). Moving up the hierarchy within a branch answers
the question, “of what more general objective is this an aspect?” (Katzer, 2002). When building
the value hierarchy, if a value cannot be decomposed into more than one lower tier value, it
should not be decomposed. As the number of tiers within a hierarchy increases, its size increases
vertically.
2.4.3.2.3 – Desirable Properties of a Value Hierarchy
Kirkwood (1997) presents five properties that are desirable for any value hierarchy.
These properties include completeness, nonredundancy, decomposability, operability, and small
size. Completeness is considered one of the most important properties for a hierarchy to exhibit.
Another way of describing completeness is if a hierarchy is “collectively exhaustive.” It stands
to reason that any hierarchy must contain all of the values important to the decision-maker. This
includes any value that is required to evaluate the fundamental objective. For a hierarchy to
show completeness, it must be possible to evaluate the objective based only on the values
presented in the hierarchy. If there are other considerations required for evaluation, they must be
added to the hierarchy. This includes all values, no matter how small of a part they may play in
the final evaluation. Their magnitude of importance is considered during the weighting phase of
the evaluation (Kirkwood, 1997).
There may also exist values that appear to be more basic to the problem than evaluation
criterion. These values may be “promoted” to “screening criteria.” Screening criteria will be
discussed in more depth in Section 2.4.3.6 - Alternative Generation. Determining the difference
between values and screening criteria may be difficult. Screening criteria should be used for the
sole purpose of reducing the number of alternatives (Kirkwood, 1997). For example, a value
may be “Distance” while a screening criteria would be “Less than five miles away.” The
distance value simply states that the relative distance is important in the end decision. The
screening criteria “Less than five miles away” specifically eliminates certain alternatives based
on their distance (Kirkwood, 1997). Each value must also pass a “test of importance.” The
decision-maker must ask whether the inclusion of a specific value will alter the outcome of the
decision problem. If the decision-maker feels that the exclusion of a specific value could alter
the best course of action, then it must be included. If the exclusion of a value will have no effect
on the outcome, then it can be left out of the hierarchy. The major caution with this “test of
importance” is the possible exclusion of a collection of independently unimportant values, but
which serve a major part in the decision when considered together. The collective importance of
any excluded values must be continually evaluated; therefore, the excluded values should not be
completely discarded, so that future iterative evaluations may be completed on them. By
conducting the test of importance at different stages in the process and with the obvious
groupings of these values, it is possible to ensure that none of the excluded values will have a
major effect on the final decision (Keeney & Raiffa, 1993).
Nonredundancy is another very important property of a value hierarchy. Nonredundancy
is also referred to as “mutually exclusivity” (Kirkwood, 1997). A hierarchy is considered
mutually exclusive if no two values in the same tier overlap in any way. Every aspect of the
evaluation criteria must relate to one and only one value. Each tier of the hierarchy should divide
the tier above, lower levels essentially composing the values above them. The property of
nonredundancy ensures that no value is double-counted and therefore receives more weight than
it deserves. Based on these first two properties, every hierarchy at its very base must be
“collectively exhaustive and mutually exclusive.” These are possibly the two most important
properties, since they ensure that everything is included and that all values required are
represented only once in the hierarchy.
Decomposability or independence refers to a value’s influence on other values
(Kirkwood, 1997). The actual scoring of any value cannot have any influence on the scoring of
another value. Decomposability ensures that there is not only a measurement for each value, but
that the measurements themselves are also mutually exclusive. This separation of measurement
ensures that the weighting of each value may be completed (Shoviak, 2001).
An operable hierarchy refers more to the utility of the tool itself. Any value hierarchy
must be understandable, at a minimum to those who must use it in an evaluation (Kirkwood,
1997). This is a rather subjective property, but in communicating a hierarchy or decision analysis
tool, the users must be able to understand quickly and easily the points that the model builder is
trying to get across. The more technical the subject matter, the more difficult it will be to satisfy
this property, although the subject matter itself does not necessarily have to have an effect on the
understandability of the hierarchy itself. If the reader is able to understand the tool, then it is
considered operable.
The last desirable characteristic of a value hierarchy is small size. In comparison, a
smaller hierarchy is generally preferable to a larger one (Kirkwood, 1997). The key tradeoff is
that it must also be collectively exhaustive. Therefore, the small size property is directly dictated
by the minimum amount of information necessary to properly evaluate the decision problem.
Small size also has an influence on the operability of the hierarchy. Larger hierarchies are
generally more difficult to communicate to stakeholders than a compact hierarchy. The size also
becomes an issue in the employment and analysis of the tool. A larger, more complex tool will
be not only more difficult to use, but more difficult to evaluate.
The size issue must be considered in two dimensions. The nature of hierarchies is both
horizontal and vertical. Therefore, the model builder must be sure to include not only the
necessary breadth, but depth as well. Breadth of the model is determined by the number of
values to which the fundamental value can be decomposed. As the level of tiers increases, the
number of values generally increases exponentially. Therefore, an increase in depth has a direct
effect on the breadth of the hierarchy. To keep these two issues under control, the “test of
importance” and guidance by the model builder must be used to ensure that each tier and value
are directly influential to the fundamental value.
2.4.3.3 Step 3 - Develop Evaluation Measures
Evaluation measures or attributes exist for determining how well an alternative performs
with respect to a particular value. This can be accomplished qualitatively or quantitatively, but
each lowest-tier value must be measurable. The evaluation measures should provide the
mechanism for turning a subjective decision into an objective decision (Kirkwood, 1997).
Graphically, the measures appear below the lowest level of decomposition in the value hierarchy.
The model builder should use as many measures as necessary to properly quantify the attributes
of the value.
2.4.3.3.1 – Types of Evaluation Measures
Evaluation measures may be classified as either natural or constructed and direct or
proxy. The type of measure depends upon the availability of data regarding the value as well as
whether the measure is qualitative or quantitative. A natural scale is one with a common
interpretation to any audience. Constructed scales are developed specifically for measuring the
value (Kirkwood, 1997). Generally, constructed scales are used when no natural scale is evident
or they may also be used when there is not enough data to measure the value exactly (Kirkwood,
1997).
In addition to being either natural or constructed, a measure will also be either direct or
proxy. Direct scales measure the degree of attainment of the value explicitly. A proxy measure
still measures the value, but does so indirectly. Proxy measurement may use some other piece of
data or a collection of data that represents the degree of attainment of the value. It is possible for
any measure to be categorized as any combination of natural/constructed and direct/proxy
(Kirkwood, 1997). A Natural-Direct measure is the most preferable, as it measures the value
most accurately. Natural-Proxy and Constructed-Direct are next in terms of desirability, and a
Constructed-Proxy scale is the least desirable since it requires interpolation between the measure
and the value (Table 2.4) (Dawley, Marentette, & Long, 2008).
2.4.3.3.2 – Desirable Properties of Evaluation Measures
Just as with values, several properties are desirable for evaluation measures. In the case
of the measures, model builders should consider measurability, operationality, and
understandability. Measurability “defines the associated value in more detail than that provided
by the value alone” (Keeney, 1992). Each measure must define the value as intended by the
decision-maker. Operationality refers to a measure’s ability to “express relative preferences for
different levels of achievement of an objective” (Keeney, 1992). Finally, a measure is considered
understandable if any audience can easily understand its purpose as was originally intended by
the model builder.
2.4.3.4 Step 4 – Value Function Creation
Value functions exist for the purpose of converting the measurement of an objective into
value units. Converting the measurements into value solves the problem of the values being
measured with different scales and different units. A Single Dimension Value Function (SDVF)
plots the measurement of the value (x-axis) versus a related value unit from zero to one (y-axis)
(Kirkwood, 1997). The least preferred score of a measurement will relate to zero, while the best
possible score earns a full value of one. An alternative’s degree of attainment of the value in
question will be plotted on the x-axis of the measure. SDVFs are built using inputs from the
decision-makers, stakeholders, or available data on the values.
SDVFs are defined by their shape and monotonicity; they may also be either continuous
or discrete. Continuous SDVFs are either monotonically increasing or monotonically decreasing.
Figure 2.5 is an example of a linear, monotonically increasing SDVF, meaning that the difference
in value between 10 and 20 on the x-axis is the same as the difference between 70 and 80.
Figures 2.6 and 2.7 are examples of exponential monotonically increasing SDVFs. In the case of
Figure 2.6, the difference in value between 10 and 20 on the x-axis is considerably smaller than
the difference between 70 and 80. This is referred to as a “convex” exponential curve. Figure
2.7 is referred to as “concave,” and exhibits similar properties as a concave SDVF. Therefore, in
this case, as more of the score is attained, the value gained gets smaller (more value is earned
early), whereas in Figure 2.6, as more of the score is attained, the value gets exponentially larger
(Kirkwood, 1997). Figures 2.8, 2.9, and 2.10 are all examples of monotonically decreasing
SDVFs; measure 5 is a linear decreasing SDVF while measures 5 and 6 are exponentially
decreasing SDVFs. (Kirkwood, 1997). The decreasing SDVFs are interpreted the same way as
increasing SDVFs.
Figure 2.11 is an example of a piecewise linear SDVF. Piecewise linear curves may also
be monotonically increasing or decreasing. They are composed of multiple linear sections that
are broken by inflection points. In the example measure, value is earned more quickly between
x-axis values of 25 to 70, value is earned more slowly when the x-axis values are smaller than 25
or greater than 70 (Kirkwood, 1997).
Figures 2.12, 2.13, 2.14, and 2.15 are examples of “S-Curve” SDVFs. “S-Curve” SDVFs
are a type of exponential curve which may also be either monotonically increasing or decreasing,
but take on the properties of a piecewise curve while retaining the exponential shape. The four
example measures shown are the four possible general configurations of S-Curves. They
account for both monotonically increasing and decreasing curves as well as concave and convex
shapes.
Figure 2.5. Monotonically Increasing Figure 2.6. Monotonically Increasing
Figure 2.7. Monotonically Increasing Figure 2.8. Monotonically Decreasing
Figure 2.9. Monotonically Decreasing Figure 2.10. Monotonically Decreasing
Figure 2.11 – Piecewise Linear Figure 2.12. Monotonically Increasing
Figure 2.13. Monotonically Increasing Figure 2.14. Monotonically Decreasing
Figure 2.15. Monotonically Decreasing Figure 2.16. Discrete
The final possible type of SDVF is discrete. In this style of value function, the possible
scores are grouped into categories or bins. The value, therefore, increases incrementally to
account for the changes in categories. This type of SDVF is particularly useful for qualitative or
binary measures. The categories must be well defined so that there is no question as to which
category an alternative belongs. Figure 2.16 shows an example of a discrete measure.
2.4.3.5 Step 5 – Value Hierarchy Weights
With the value hierarchy and SDVFs created, the decision-maker has a solid frame of
reference for what is important, as well as a basis for the values implicit in the decision. Each
value must then be weighted to show its relative importance to the decision-maker. There are
two primary methods for determining the weight of each value, the direct weighting method and
the swing weighting method. Both of the methods give rise to local weights and global weights
(Shoviak, 2001).
Local weight refers to the relative importance of a single value in relation to other values
in the same branch and tier. Therefore, the values in each branch and tier must sum to one. In
the case of Figure 2.5, all values in Tier 1 must total 1. Therefore, Tier 1 Value 1 may have a
weight of 0.6, while Tier 1 Value 2 has a weight of 0.3 and Tier 1 Value 3 has a weight of 0.1.
This means that Tier 1 Value 1 is twice as important as Tier 1 Value 2, and Tier 1 Value 2 is 3
times as important as Tier 1 Value 3. This method is applied to each tier and branch of the
hierarchy. Therefore, the weight of Tier 2 Value 1 and Tier 2 Value 2 must also sum to 1 to make
up the total value of Tier 1 Value 1. Measures are also weighted in the same way. As the process
moves, the model builder and decision-maker may weight the hierarchy moving from the lowest
tier to the highest or from the highest to the lowest.
One method for leading the decision-maker to a conclusion of the weights is the use of
the “100 Coin” method (Jurk, 2002). In this situation, the decision-maker is asked to distribute
100 “coins” between the values; i.e. if the decision-maker had 100 coins to distribute between the
different values, where would they be placed? In this method, the number of “coins” placed on
any value becomes the percentage of importance or the percentage of emphasis placed on one
value when compared to others in the same tier and branch. Decision-makers may also be asked
to rate each value relative to the others. For example, the decision-maker may say that “Tier 1
Value 1 is twice as important as Tier 1 Value 2 and Tier 1 Value 2 is 3 times as important as Tier
1 Value 3.” In this case, the weights become 0.6, 0.3, and 0.1, respectively. Ratios may also be
used to determine the weights.
Local weights determine the relative importance of values in relation to the other values
on the same tier, but values must also be rated in terms of “Global” importance. These weights
are referred to as “Global Weights.” The global weight may be found through direct weighting or
it may be found after local weighting by their multiplicative functions in relation to the overall
fundamental value in the hierarchy. Global weights must sum to 1 across an entire tier as
opposed to local weights, whose sum must be one for a tier in any given branch. Consider a case
in which Tier 2 Value 1’s local weight is 0.75 and Tier 2 Value 2’s local weight is 0.25 locally.
In this case, the weights are multiplied up the hierarchy to determine their global importance. If
Tier 1 Value 1’s weight is 0.6, then the global weights for Tier 2 Value 1 and Tier 2 Value 2
become 0.45 (x = 0.75*0.6) and 0.15 (x = 0.25*0.6), respectively. Figure 2.17 shows the
example hierarchy with local weights displayed and Figure 2.18 shows the hierarchy with global
weights displayed. As is evident here, all values in a tier total 1 and the measures are then
weighted according to the value that they measure.
Figure 2.17. Example Hierarchy with Local Weights
Figure 2.18. Example Hierarchy with Global Weights
Another method of weighting a hierarchy is known as “Swing Weighting.” This method
is a local weighting technique and was compiled from procedures set forth by Chambal (2008)
and Kirkwood (1997). This technique examines the possible outcomes that may be reached
based on the weights of the values. The decision-maker is asked to examine each tier of values
and determine the change in increments of value that would be reached by varying the weight of
each value from its least preferred state to its most preferred state. These increments are then
placed in increasing order and assigned a factor of importance in relation to the smallest value.
These increments, which should sum to one, are then solved as a system of equations to
determine the local weight within the given tier (Jurk, 2002).
2.4.3.6 Step 6 – Alternative Generation
One of the advantages of using VFT is the ability to generate alternatives as opposed to
simply choosing from given alternatives. Once the hierarchy has been weighted, this is possible.
In the initial stages of alternative generation, experience gained by simply creating the hierarchy
will often yield a great number of possible alternatives. Building the hierarchy often gives the
decision-maker new ideas and insights into the importance of the outcome and new ideas for
alternatives. “Either the alternatives are somewhere in the mind waiting to be found, or they can
be created from what is in the mind” (Keeney, 1992, p. 198).
If too many alternatives are found, the list must be reduced to a manageable number. In
this case, additional screening criteria may be added to eliminate some of the less desirable
options. Screening criteria are based on values that serve to eliminate some alternatives prior to
scoring. A screening criteria may be established if some alternative scores zero on a particular
measure. Screening criteria may also be something that is required by the decision-maker; i.e., if
some value or condition is not true, the alternative is eliminated completely from consideration.
Some values may be so important that an alternative will not be considered without their
inclusion. Alternatives may also be eliminated based on known values. If there are not enough
alternatives, this usually suggests a gap in the value hierarchy, i.e., there is something important
which is not being considered and that would give the decision-maker more alternatives.
Strategy generation tables may also be used to generate alternatives (Kirkwood, 1997). The most
important thing to remember during this process is that the alternatives must satisfy the values in
the hierarchy. In some cases, alternative generation may not be necessary if the field of
alternatives is given or if some outside factor limits the alternatives.
2.4.3.7 Step 7 – Alternative Scoring
Following alternative generation, each alternative must be individually scored. Data is
collected regarding each alternative and its attainment of each lowest-tier value (based on the
measures of those values). Scores are then assigned to each measure within each alternative.
During this process, the y-axis or value units are hidden from the scorers, so that the value does
not impact the scoring of the alternatives. Each score must be well documented, clearly defined,
and repeatable by anyone who scores the alternative.
2.4.3.8 Step 8 – Deterministic Analysis
The deterministic analysis step combines all data collected to this point. Through the use
of an additive value function, the scores given to each alternative (step 7) are converted to value
units (step 4), and then multiplied by their weights (step 5) to yield a single aggregate score. The
additive value function is the way in which the decision-maker may perform detailed analysis of
the alternatives (Shoviak, 2001). The general additive value is described in equation 2.1
(Kirkwood, 1997):
(2.1)
Where v(x) is the overall score of the alternative, is the value of the score on the ith measure,
weight of the ith measure, n is the total number of measure, and the sum or all must equal
one.
2.4.3.9 Step 9 – Sensitivity Analysis
Sensitivity analysis is performed on the hierarchy to provide additional insight into the
weighting of the values and how they affect the scores of the alternatives. Typically, sensitivity
analysis is performed on the higher tiers of the hierarchy, since altering the weights of values on
lower tiers will have less effect on the total score. Sensitivity analysis is performed by
systematically altering the weight (local or global) of one value, while keeping the other weights
on that tier proportional. The weights must continue to sum to one across a tier. Sensitivity
analysis serves to answer the question, “How would this decision change if another interested
party had weighted the hierarchy or provided data for the SDVFs?” (Katzer, 2002, p. 46).
2.4.3.10 Step 10 – Recommendations Presentation
The final step in the process requires the model builder to present recommendations to
the decision-maker. Parnell suggests that one-third of decision analysis efforts should be placed
in the recommendations presentation. The recommendations must be easy to understand for all
audiences. They must also explain the decision made and why it was made. It is important to
remember that the final decision still lies in the hands of the final decision authority. The VFT
process serves to assist the decision-making process and provide objective data and an analysis
of alternatives. There may be cases where the recommended alternative may not be chosen.
2.5. Management and Planning Tools
As a part of the value determination process, it may be necessary to organize information
found during the document review phase. Several tools exist for managing ideas and concepts.
Tague (2004) identified seven management and planning tools; these consist of affinity diagrams,
interrelationship diagrams, tree diagrams, prioritization matrices, matrix diagrams, process
decision program charts, and activity network diagrams. These tools were organized in 1976 in
an effort to collect quality control techniques (Tague, 2004). All of them were not created at this
point, but were put together in a single work for managers to easily locate. The tools allow
managers to organize ideas and concepts to make better, more efficient decisions, which take into
account all known information.
2.5.1 Affinity Diagrams
The affinity diagram, developed by Jiro Kawakita in the 1960s, was created to expound
on the brainstorming group creativity technique. In brainstorming, groups of people come
together and generate as many ideas as possible related to a single concept. This method focuses
on the power of the group to generate a larger quantity of ideas than any individual can (Osborn,
1953). Affinity diagramming takes this more generalized approach and improves upon it. The
brainstorming process is used to initially generate ideas, but through a process of organization
and idea mapping, combined with subsequent discussion, the ideas are eventually sorted into
descriptive groups. Affinity diagrams are used in situations when there are a great many ideas or
issues, which are in no apparent order and complex in nature (Tague, 2004).
The decision-maker benefits from the large quantity of ideas generated by the team by
using the affinity diagramming technique. The process begins by describing the problem and
ensuring that all team members are familiar with the issues. The brainstorming technique is then
used to gather as many ideas as possible. These ideas are then written on individual note cards or
“sticky notes.” The members of the affinity diagramming team then physically begin to silently
organize the notes into groups. Each person in the process may have their own ideas, so some
notes are moved several times, but no discussion is allowed during this process. Once all notes
are placed in groups or set aside for discussion, the resulting groups are discussed and examined
by the team. Finally, any “super groups” that have emerged are created by defining the
individual groups and further organizing the cards. The end result of this process is a number of
groups and possibly hierarchies which describe ideas (Tague, 2004).
The Value-Focused Thinking process involves a top-down analysis approach, but at
times, it is difficult to determine the lowest-level tier values. Affinity diagramming provides a
method for combining a bottom-up approach to the existing process to ensure accurate definition
of the lowest tier (Pruitt, 2003). Affinity diagramming is an appropriate technique, due to its
ability to organize large amounts of complex information into groups with sub-categories being
built into the technique.
2.5.2 Other tools
In addition to affinity diagramming, there are several other tools commonly used in the
management and planning industry (Tague, 2004). Each of these tools is used for a very specific
purpose. Interrelationship diagrams, for example, describe the links and interfaces between
ideas. They serve to identify any cause and effect relationships that exist. They are also used for
complex issues, but are generally used as a follow-on to affinity diagramming when cause and
effect relationships must be defined (Tague, 2004). Tree diagrams may also be used to
breakdown more general ideas into their components. Tree diagrams often depend on affinity
diagrams to first identify the issues upon which to expand (Tague, 2004). Matrix diagrams are
another way of showing relationships between ideas, but they organize the relationships
differently than an interrelationship diagram. Matrix diagrams can also show relationships
between multiple groups of information, while including specific information regarding the
relationship. They can be categorized into six different “shapes,” which can be used for different
numbers of groups and different types of relationships (Tague, 2004). Matrix data analysis then
allows the decision-maker to perform complex mathematical analyses on the resulting matrices
(Tague, 2004). “L-Shaped” matrices are often used to prioritize ideas (Tague, 2004). An arrow
diagram; also known as program evaluation and review technique (PERT) chart, network
diagram, activity chart, critical path method (CPM), or node diagram; is used to describe the
order of tasks. Arrow diagrams are very useful in showing chronological order of ideas. They
can describe when tasks precede others as well as durations (Tague, 2004).
2.6 Net-Centricity
Net-centricity refers to the process by which several nodes in communication with each
other operate. In the evolving technological world, it is increasingly important that the complex
network of personnel, devices, services, and information be connected. The speed of
communication and efficiency of passing information between this complex system of nodes has
a major part in the decision-making process. The information age has given society access to a
large amount of previously unavailable information; it is now a matter of ensuring that the
technology and infrastructure exists to move the information to the correct person at the correct
time.
Net-centricity has given way to Net-Centric Warfare (NCW) within the U.S. military.
NCW is defined by Albert et al. (2000) as,
an information superiority-enabled concept of operations that generates increased
combat power by networking sensors, decision-makers, and shooters to achieve
shared awareness, increased speed of command, higher tempo of operations,
greater lethality, increased survivability, and a degree of self-synchronization.
The concept of NCW is the idea of linking nodes to transfer information, thereby ensuring
information superiority for the warfighter (Alberts, Garstka, & Stein, 2000).
2.6.1 Net-Centric Enterprise Architecture
As systems architecture grows, the importance of capturing the communication between
nodes in a relevant manner is becoming increasingly important. To emphasize this idea,
NetCentric Enterprise Architectures are becoming the standard in the systems engineering field.
A net-centric enterprise architecture is formally defined by Nzuwah (2003) as,
a light-weight, massively distributed, horizontally-applied client/server
architecture, that distributes components and/or services across an enterprise’s
information value chain using internet technologies and other network protocols
as the principal mechanism for supporting the distribution and processing of
information services.
The concept of a net-centric architecture, therefore, is any architecture that makes use of
technology to ensure the proper communication of all nodes in the system. In the case of
systems architecture, this can refer to not only the products themselves, but to the development
of the products and the net-centricity of the instantiated system. The system being represented
by the architecture must also hold to the principles of net-centricity.
2.6.2 Net-Centric Enterprise Solutions for Interoperability (NESI)
Currently, the DoD does not explicitly require the implementation of net-centricity;
however, it has made the intended direction to move toward it obvious. The DoD’s Chief
Information Officer (CIO) has stated a goal to integrate data into a central network and change
the paradigm from data “push” to “pull” (Department of Defense Chief Information Officer,
2003). To this end, the U.S. Navy Program Executive Office for Command, Control,
Communications, Computers, and Intelligence, in cooperation with the U.S. Air Force Electronic
Systems Center and the Defense Information Systems Agency has produced the Net-Centric
Enterprise Solutions for Interoperability (NESI) as a series of guidance documents. NESI
provides cradle-to-grave actionable guidance for the implementation of net-centric systems that
meets the goals set forth by the DoD CIO. NESI pulls together several sources to provide a body
of knowledge encompassing architectural and engineering information for each step of the
acquisition process. In addition to the general guidance, NESI contains checklists for the project
manager to ensure compliance with the guidelines set forth in NESI (US Navy Program
Executive Office for Command, Control, Communications, Computers, and Intelligence, 2008).
Unfortunately, at this point, compliance with NESI is not required by the Navy or any DoD
agency (Eitelberg, 2008).
Chapter 3. Methodology
The Joint Force Protection Advanced Security System (JFPASS) has the unique challenge
of creating a single, joint architecture to represent force protection across the services. This
architecture must be understood by various stakeholders as well as represent an effective system,
which will be the groundwork for all future force protection acquisition efforts. This led the
architecture developers to seek out a tool for evaluating and gaining insight into their product.
Drawing upon Value-Focused Thinking (VFT), this research presents a Value-Driven
Enterprise Architecture score for mapping architectural products to stakeholder acquisition
values. The generalized VFT methodology laid out in Chapter 2 is built upon in this section to
extend to enterprise architecture evaluation with a value focus in application to the JFPASS
architecture. Each step of the process is examined in depth to build the final hierarchy. This
includes all steps up to and including step 7.
3.1 Problem Identification
The JFPASS project grew out of the DoD’s need to be both more net-centric and more
joint. A series of architectural products were developed to meet this requirement. This
architecture has similar problems to other architectures in the lack of effective evaluation tools,
but a new element for this problem is the extreme complexity of the architecture and the desire to
examine both the architectural quality and the System Effectiveness with a single tool.
To solve the problem of evaluating the architecture, the research question was framed as:
“How should common Joint Force Protection values be used to evaluate a “To-Be” architecture
for net-centric force protection” (Havlicek, 2008). In this case, the system is the JFPASS
architecture. To further define the problem, the context was first researched and defined.
Several documents refer to the protection of personnel, assets, and information as the three key
areas to be protected in the context of joint force protection (IUBIP, 2006; JCS, 2004; Office of
the CJCS, 2008; Office of the CJCS, 2007). Figure 3.1 shows a hierarchy of the documentation
and guidance that guides force protection. Figures 3.2 through 3.4 specialize this idea to show
the mediums from which each area must be protected.
Figure 3.1. Documentation Hierarchy for Force Protection
Figure 3.2. Protect Personnel Specialization
Figure 3.3. Protect Assets Specialization
Figure 3.4. Protect Information Specialization
The National Security Strategy (NSS) provides the highest-level guidance for military
operations (Office of the President of the United States of America, 2002). The National
Military Strategy (NMS) specifies the military portion of the NSS (Joint Chiefs of Staff, 2004).
One of the four tenets of the NMS is force protection, which is divided into the protection of
personnel, assets, and information. The IUBIP specifies that the scope of force protection be
across fixed, semi-fixed, and mobile sites. Fixed sites are defined as those facilities in either the
Continental United States (CONUS) or Outside the Continental United States (OCONUS) where
Mutual Security Agreements or Status of Forces Agreements exist. Semi-fixed sites are any
locations established for a temporary purpose, which includes expeditionary locations or
locations in the CONUS or OCONUS that are no intended to be occupied for more than one year
at construction. Finally, mobile sites are those where a unit is performing its mission, including
convoys, logistics patrols, or other movements between sites.
At each of the three types of sites, personnel and assets must be protected in the same
basic ways, as the personnel generally depend on the assets (vehicles and buildings) for shelter
and movement. Often attacking an asset will have a direct affect on the security of the associated
personnel. Chemical, Biological, Radiological, Nuclear, and High-Yield Explosives (CBRNE)
risks, however, are only considered for personnel, since CBRNE threats do not have an effect on
the assets, aside from denial of their use. The CBRNE threat is intended to impact the personnel
occupying the asset. Protecting information is a slightly different concept. Information must be
protected in two contexts: the infrastructure that carries information and the access that
individuals have to that information. Access control involves not only electronic access control
such as ensuring that only authorized personnel have access to the information, but also ensuring
that access is not granted from person to person-to-mission critical information.
Within the context of protecting personnel, assets, and information, force protection must
accomplish all of the DAWDR (Detect, Assess, Warn, Defend, Recover) activities. For the
JFPASS project, the focus of the architecture is only on the Detect, Assess, and Warn activities,
although mechanisms exist in the architecture for the Defense and Recovery of a location.
3.2 Create Value Hierarchy
The first decision to be made regarding the value hierarchy was the basic split of how to
evaluate the system. Two divisions of quality must be addressed: the quality and accuracy of the
architectural views or products and the effectiveness of the instantiated system that the
architectural products attempt to represent. Due to the complexity of the system, the divisions
were separated into separate branches. This decision leads to better decomposability and easier
operability (Kirkwood, 1997). Splitting the architecture from the system also ensures
independence and mutual exclusivity of each value, as some ideas apply to each side, but in a
different context. Furthermore, the operability of the hierarchy is improved by making the
hierarchy not only easier to read, but also extendable to other situations. With the architecture
and system aspects separate, both branches may be used independently in other projects. Either
of the two branches may also be replaced by another branch to increase accuracy for use on
another system, thus making the hierarchy modular. However, making this separation violates
the desirable property “small size” of a hierarchy. By separating the two values, the hierarchy
would potentially be larger in terms of total branches, although the number of measures would
stay the same due to the requirement to measure the same information, regardless of the outcome
of this decision.
3.2.1 Hierarchy Background
The initial value hierarchy created to address the problem presented in Section 3.1 was
developed using “ilities” and the affinity diagramming process. A representative list of ilities
was gathered from various sources, including Ross (2006); McManus, Richards, Ross, and
Hastings (2007); and INCOSE (International Council on Systems Engineering, 2007). The
internet website “Wikipedia” also contains a list of ilities, which was used to ensure that as many
ilities as possible were gathered. Since Wikipedia is a user-edited site, this gives a wider pool of
quality attributes from which to pull at the risk of getting inaccurate information. Since this data
pull was intended only to gather terms, not definitions or uses, inaccurate information was not an
issue (Wikipedia, 2006). This search for ilities was done in place of an on-site brainstorming
process, to ensure that previous research and uses of the quality attributes were represented in the
list. The full list of 98 ilities is shown in Appendix A. Ilities were chosen for this exercise for
their historical use in describing the quality attributes of various systems. By finding all of the
applicable ilities related to the project, it was possible to capture all of the necessary quality
attributes to describe both the architectural quality of the products and the effectiveness of the
instantiated system.
In accordance with the affinity diagramming process, each of these ilities was written on
individual note cards. The affinity-diagramming team was then sequestered in silence to
physically arrange the note cards into groups of similar qualities. At the conclusion of the
process, 30 subgroups were found. This led to an interactive discussion of the groupings and
further refinement of the subgroups. This discussion first identified quality attributes and
subgroups that did not apply to the JFPASS project. The eliminated ilities were: composability,
demonstrability, learnability, nomadicity, portability, predictability, seamlessness, testability,
timeliness, trainability, and transactionality. Composability refers to creating some new form by
combining components. While the construction of the system will be created through the
combination of its components, the ability to do so was not considered a measurable quality
attribute. The actual combination of components is a design consideration that must be
considered before any architectural products are produced. Demonstrability and testability refer
to the ability of the system to be demonstrated. These ilities were eliminated because the
JFPASS Joint Capabilities Technology Demonstration is already in progress; therefore, the
demonstrability of the system is assumed and is not required as a quality attribute. Learnability
and trainability were considered too vague due to the confusion as to whether the learning
referred to the system or the system users. Trainability is an attribute that will be considered at
some point in the creation of the system, but as JFPASS is a “system of systems” still in the
design phase, there is no way to definitively measure the ability of the system to be taught to
others. Nomadicity and portability were eliminated because the JFPASS is not considered a
“mobile” system. Predictability and seamlessness were also vague in definition in terms of the
JFPASS system. Timeliness refers to the time required to create the system. While timeliness
could be defined to show when the system will be fielded and implemented, there are currently
no time estimates or requirements for the fielding of the system. Finally, transactionality was
eliminated due to the connotation of its root word, “transaction.” There will be no monetary
transactions taking place as a function of this system; therefore, the ility was eliminated. The
resulting 22 subgroups were then examined for agreement and accuracy. After minor alterations
to the locations of certain words based on definition, super groups were formed. The two
primary supergroups were based on the decision to split the architecture and system qualities.
The groups related to the quality of the system being represented were placed in a supergroup
called “System Effectiveness,” while the groups related to the quality of the architectural products
were placed in a supergroup called “Architecture Quality.” Architecture Quality is addressed by
Cotton and Haase (2009). The System Effectiveness supergroup is addressed here. The group
names shown in the following discussion and tables are based on the final decisions, following
sponsor discussion.
The System Effectiveness value was subsequently decomposed into three second-tier
values: Capability, Maintainability, and Interoperability. Maintainability was comprised of two
third-tier values called Dependability and Resiliency. Dependability was further decomposed
into Supportability and Reliability, while Resiliency was decomposed into Survivability and
Recoverability. Appendix B shows the final System Effectiveness quality attributes grouped
according to value, along with their synonyms.
Prior to creating the hierarchy itself, each value group name was defined to ensure that
the synonyms that compose the group were accounted for in the final consideration of the group.
Each of the value definitions are listed in Table 3.2. These definitions incorporated the
definition of the quality attribute itself and the synonyms for the particular group name, as well
as the value to the decision-maker. Following sponsor discussion, the defined and grouped ilities
were converted into a value hierarchy. The resulting hierarchy is shown in Figure 3.5
Table 3.2 System Effectiveness Value Definitions
uoyesiunwiwo
Aqyiqesadouayu|
AYOIVIOH
SNVA,
SSOUSATIOOJTA
WI9}sAS
“S'¢
DUNST
Ayyiqesaaooay
Aqyiqeaiains
Ayiqeray
Aqiqeyoddns
Yyo4s9}u|
Adualjisay
Ayiqepuadaq
Aqyeonoeid
ssaujnjasoding
Ayiqeurequieyy
SSOUBANIANA
wa
ss
Aujiqedep
3.2.2 System Effectiveness Hierarchy
In the creation of the value hierarchy, several naming changes, definition changes, and
alterations to structure were required to create a representative hierarchy. These changes were
accomplished through further literature review and during meetings with the decision-maker and
sponsoring organization. The value hierarchy was created using the resulting groups of the
affinity diagramming process, based on quality attributes. This section describes the values
found on each level of the hierarchy in more depth. The initial draft hierarchy was presented
during these meetings, resulting is discussion and ultimately validation by a panel of Subject
Matter Experts (SMEs) and the decision-maker.
In the initial iteration of the value tree, the System Effectiveness branch was called
System Value. During a discussion regarding the naming and definition of the Capability value
(which was initially named Effectiveness), it was decided that the term Effectiveness better
described the entire System branch as opposed to a single value under the system branch;
therefore, Effectiveness was promoted to the branch name to better describe all of the values
under the branch. Each of the values in the branch (Capability, Maintainability, and
Interoperability) relate to the overall effectiveness of the instantiated system.
3.2.2.1 Capability Branch
Capability was originally known as Effectiveness. However, Effectiveness was decided
to be too broad of a term to describe the purpose of the Capability branch. The Capability
branch was defined as, “A System’s ability to produce the expected or desired results on the
battlefield.” Subsequently, it is intended to represent the operational ability of the system to
accomplish its intended purpose. In other words, a system must have the ability to meet the
objectives for which it was designed. On the third tier of the hierarchy, Capability is
decomposed into Purposefulness, Practicality, and Flexibility. These lower tier values serve to
specify the ideas that make up Capability.
Within the context of Capability, the “the ability of a system to address the problem
which it is intended to solve or the relevance of a system in a given context or situation” was
called the Purposefulness of the system. This value actually defines whether the system does
what it is intended to do in the proper situations. The value of Purposefulness is used to account
for a major idea of System Effectiveness. This value accounts for a great deal of the Capability of
a system.
The Practicality of the system defines whether it can actually be realized. Practicality is
officially defined as, “The system’s ability to be achieved within realistic constraints, including
economic, constructability, and timeliness.” This value was considered important due to the
inability of the system to accomplish its intended objectives if it cannot be constructed or
implemented within realistic constraints. Without a practical system, the goals of the system
designer will not be achieved.
The Flexibility value was initially placed under the Interoperability branch due to a
connotation involving its ability to change to operate with other systems. However, a system’s
ability to change in relation to other systems is primarily determined by its initial design once it
is implemented. Therefore, Flexibility was moved to better capture the system’s ability to change
to meet changing and evolving operational objectives. This allowed a more strict definition,
which eliminated the problem of broad connotations for the word Flexibility. The official
definition of Flexibility is “the ability of a system to be changed based on operational need. This
changeability refers to its ability to be altered before, during, and after a conflict.”
3.2.2.2 Maintainability Branch
The entire Maintainability branch was validated by the decision-maker as being
acceptable as initially presented. Definitions of values were altered slightly to ensure maximum
achievement of the decision-maker’s values. Maintainability itself is defined as, “A system’s
ability to be kept at its intended level of operation.” Any system requires some type of regular
action to ensure that its intended operation continues uninterrupted.
Below Maintainability, the first branch was called Dependability. Dependability is
officially defined as, “A system’s ability to continue operating at its intended standard.” The
Dependability branch refers to the maintenance of the system under normal operating conditions.
Dependability deals with the “peacetime” operations and maintenance of a system. On the next
tier, Supportability is one of the values under Dependability. This value deals with a system’s
ability to operate as normal, given a standard maintenance schedule. Dependability’s definition
is “the ability of a system to be realistically sustained and remain functional and useful given the
expenditure of a reasonable amount of effort.” A reasonable amount of effort refers to operations
performed in accordance with a standard maintenance schedule. This value does not incorporate
major alterations or repairs, only a normal recurring work program type of maintenance. The
second value under Dependability is Reliability. The Reliability value deals with the ability of
the system to continue its operation if maintained properly. Reliability is the relationship
between Supportability and the operation of the system. Its definition is “the ability of a system
to perform as intended and execute given functions if properly maintained and supported.”
Resiliency is the second branch falling directly under Maintainability. Resiliency is “a
system’s ability to be returned to its intended standard.” If the Dependability branch deals with a
system’s operation during normal peacetime or uninterrupted operations, Resiliency refers to the
system’s operation following some type of interruption. In the context of Joint Force Protection,
this interruption is some type of hostile action. Resiliency measures how easily a system may be
repaired following such an action. Under Resiliency, the first of two values is Survivability.
Survivability is the part of Resiliency dealing with a component’s ability to withstand some
hostile action. It is defined as “the ability to survive attack or other enemy action and continue to
operate as originally intended or retain the ability of being repaired and restored to operational
status.” Survivability measures how a system operates once it has been affected by some hostile
action. The second Resiliency sub-value is Recoverability. Recoverability is another portion of
Resiliency referring to a system’s ability to be returned to full operational status following an
interruption of operations due to hostile action. It is defined as “the system’s ability to be
repaired or recovered following an attack or other damage within an allotted time frame.” The
definition refers to repair and recovery, both of which are intended to allude to the returning of
the system to its original intended operation or the state that it was at prior to the hostile
interruption. “An allotted time frame” is a time period at the decision-maker or user’s discretion.
Any system or system component must be designed to be returned to its original level of
operation within a specified time frame.
3.2.2.3 Interoperability
Interoperability is the value which measures the ability of the system to operate in
conjunction with other systems and nodes. Interoperability covers both the net-centricity of a
system and its ability to be used in different contexts. Interoperability is defined as “a system’s
ability to be applied within different contexts, including other services and organizations.”
Interchangeability is the portion of Interoperability that accounts for the system and components
to be useful across different contexts. The decision-maker and subject matter experts felt that it
was very important for components to be interchangeable. Each system and component should
be able to be changed out for another seamlessly. This concept includes personnel as well as
components. Soldiers, Airmen, Sailors, and Marines must all have the same basic knowledge on
the systems in question and be trained to the same level in force protection awareness. JOINT
OPERATIONS are extremely important and to accomplish a truly Joint environment, all personnel
must have a similar training base. Interchangeability is defined as “a system’s ability to be
applied within different contexts, including other services and organizations.”
Communication refers to the ability of the nodes within the system to communicate with
each other. Both infrastructure and common languages are important for this value to be
achieved. Communication is “the system’s ability to transmit information in timely and accurate
way as to facilitate analysis, decision making, and decisive action.” An interoperable system
must send information between nodes quickly and with complete data integrity.
3.3 Develop Evaluation Measures
With the full value hierarchy built, each lowest-tier value must be measured. Therefore,
the lowest-tier values were assigned one or multiple evaluation measures. The goal of an
evaluation measure is to determine the level of attainment of each value. Table 3.3 lists all
evaluation measures, including their important characteristics. Although weights are not
discussed until section 3.3, Table 3.3 also shows the global weights (λ) for each measure. The
source for each measure suggests where a scorer should start investigating the architecture to
find the information. Table 3.4 presents the definitions for each measure. Given the vast range
Table 3.3. System Effectiveness Evaluation Measures
Measure Name
λ
Type
Source1
Lower
Bound
Upper Bound
1
OPERATIONAL NEEDS
0.041
Constructed -
Direct
SV-5
0%
100%
2
THREAT DETECTION
0.041
Constructed -
Proxy
OV-5
No
Yes
3
THREAT ASSESSMENT
0.041
Constructed -
Proxy
OV-5
No
Yes
4
WARNING PLAN
0.041
Constructed -
Proxy
OV-5
No
Yes
5
TECHNOLOGICAL AVAILABILITY
0.02
Natural -
Direct
SV-7,8,9
TRL 1
TRL 9
6
ENVIRONMENTAL IMPACT
0.02
Constructed -
Proxy
AV-1
Cannot be
built
Within all
constraints
7
MONETARY PRACTICALITY –
INITIAL
0.02
Natural -
Direct
AV-1
Over
budget
Under budget
8
MONETARY PRACTICALITY –
MAINTENANCE
0.02
Natural -
Direct
AV-1
Over
budget
Under budget
9
ADAPTATION
0.027
Constructed -
Proxy
SV-8
Static
Easy, On-Site
10
SUPPORTABILITY
REQUIREMENTS
0.035
Constructed -
Direct
SV-7
No
Yes
11
RELIABILITY REQUIREMENTS
0.064
Constructed -
Proxy
SV-7
No
Yes
12
SYSTEM REDUNDANCY
0.04
Constructed -
Direct
OV-6
None
All/Multiple
13
RECOVERABILITY
REQUIREMENTS
0.026
Constructed -
Direct
SV-7
No
Yes
14
JOINT OPERATIONS
0.033
Constructed -
Proxy
AV-1
No
Yes
15
NESI DEVELOPMENT
0.066
Constructed -
Direct
TV-1
No
Yes
16
NESI EVALUATION
0.066
Constructed -
Proxy
TV-1
No
Yes
Total of System Effectiveness Global
Weights
0.600
1. Primary source of information; other views may be required
Table 3.4. Measure Definitions
System Effectiveness Branch
Value
Measure
Measure Definition
Purposefulness
OPERATIONAL
NEEDS
What percentage of operational needs are addressed
by the system?
Purposefulness
THREAT
DETECTION
Has a Threat Detection Plan been established?
Purposefulness
THREAT
ASSESSMENT
Has a Threat Assessment Plan been established?
Purposefulness
WARNING PLAN
Has a base warning plan been established?
Practicality
TECHNOLOGICAL
AVAILABILITY
What is the Technological Availability of the system?
Practicality
ENVIRONMENTAL
IMPACT
Can the system be realized within Environmental
Constraints?
Practicality
MONETARY
PRACTICALITY -
INITIAL
Can the system’s initial cost be realized within current
budgetary constraints?
Practicality
MONETARY
PRACTICALITY -
MAINTENANCE
Can the system be maintained within current budgetary
constraints?
Flexibility
ADAPTATION
How well does the system adapt to changing threats?
Supportability
SUPPORTABILITY
REQUIREMENTS
Have supportability requirements been accounted
for?
Reliability
RELIABILITY
REQUIREMENTS
Have reliability requirements been accounted for?
Survivability
SYSTEM
REDUNDANCY
The degree to which critical systems are redundant?
Recoverability
RECOVERABILITY
REQUIREMENTS
Have recoverability requirements been accounted
for?
Interchangeability
JOINT
OPERATIONS
Have CONOPs been constructed to account for all
organizations?
Communication
NESI
DEVELOPMENT
Was NESI Guidance taken into account when
constructing architecture?
Communication
NESI EVALUATION
Has a NESI evaluation been completed on the
architecture?
of available products and latitude for which architects may use the products, it may be necessary
to examine other products or views to find the information necessary for each measure. In cases
where a specific view has been created for the sole purpose of representing a certain type of
information, that view will be considered essential in the measurement of the value at hand. This
accounts for information that may be extrapolated by the user examining several other products,
but was not explicitly stated by the architect, when it is possible.
3.3.1 Capability Measures
The goals of the measures under Capability are to collectively measure the attainment of
the Capability value. Capability is meant to determine if the system is able to produce the
sponsor’s desired effects on the battlefield. Through use of the Purposefulness, Practicality, and
Flexibility values, the three major aspects of Capability are captured. Each of these lower tier
values must be measured to determine their effect on the Capability value.
3.3.1.1 Evaluation Measures for Purposefulness
Purposefulness requires more than one measure to completely determine its level of
attainment. Four total measures score four separate aspects of Purposefulness as determined by
the sponsor. Each of these measures serves to determine a particular portion of the
Purposefulness value. This is where the Detect, Assess, Warn, Defend, Recover (DAWDR)
construct was considered.
3.3.1.1.1 OPERATIONAL NEEDS
The first evaluation measure under the Purposefulness value determines whether the
system designers have accounted for the sponsor’s initial requirements. The measure asks:
“What percentage of the Operational Needs is addressed by the system?” It is possible to trace
each operational need to a requirement, to a capability, and ultimately down to a function or
system capability. The DoDAF facilitates this primarily through the SV-5 product. The SV-5
alone is not capable of accomplishing this measurement, due to its lack of an explicit list of
OPERATIONAL NEEDS. The AV-1 product must be used in coordination with the SV-5 to find this
association. This measure is Constructed-Direct. The SV-5 is a constructed product, through
which the scorer may determine if OPERATIONAL NEEDS have been met by system components.
The measurement scale is not a common way of determining whether
OPERATIONAL NEEDS have been met, as SV-5s require some architecture knowledge to read. It
directly measures the value due to the fact that determining if all operational needs have been
accounted for determines the exact level of attainment of the value.
3.3.1.1.2 THREAT DETECTION
The THREAT DETECTION measure grew out of the DAWDR (Detect, Assess, Warn,
Defend, Recover) construct. For the system to meet its purpose and accomplish its goals on the
battlefield, it must account for at least the first three aspects of the DAWDR construct. THREAT
DETECTION measures whether a Threat Detection Plan has been developed. At this stage of
development, it is impossible to measure the actual quality of the Threat Detection Plan; it is
only possible to measure its existence. At a later time in system development, it may be possible
to alter this measure to account for the quality of the Threat Detection Plan. The activities
associated with the Threat Detection Plan will be located in the OV-5 if it is available. By
tracing the system’s activities, it will be possible to determine if there are system activities that
account for threat detection. If a Threat Detection Plan exists, the activities that accomplish it
must also be present in the OV-5. If the OV-5 is not available, the OV-1 and OV-3 will be
examined for a detection plan. This measure is a Constructed-Proxy measure. It will be obvious
to any reader if the plan exists, making the scoring scale binary. The measurement question,
however, was constructed for the purpose of this evaluation. By using the activities associated
with the plan as opposed to the plan itself, the scorer must make an inference that if there are
activities, then the plan has been created; therefore, it is a proxy measure.
3.3.1.1.3 THREAT ASSESSMENT
THREAT ASSESSMENT is similar in all ways to THREAT DETECTION, except it measures
whether a plan exists to assess the threat once it has been detected. This accounts for the second
aspect of the DAWDR construct. This measure will also examine the OV-5 to look for activities
related to Assessment (with OV-1 and OV-3 as secondary views). THREAT ASSESSMENT is a
Constructed-Proxy measure.
3.3.1.1.4 WARNING PLAN
WARNING PLAN is the third aspect of the DAWDR construct measured by this model. It
is also very similar to THREAT DETECTION and THREAT ASSESSMENT. In this case though, the
OV-5 is examined for any mentions of the activities related to warning the base population. A
warning plan should warn not only the base population, but also specific organizations as
required. Warnings may also be “tiered,” so that only certain people are warned depending on
the purpose for the warning. Warning plans may be very prevalent in the OV-1 and OV-3 in this
case, as the warning of people relates directly to both the system boundary and node
communication. It is also Constructed-Proxy.
3.3.1.2 Evaluation Measures for Practicality
The value of Practicality must also be measured using more than one aspect. Since it is
also a very complex issue and the definition calls for several layers of practicality, each of those
layers must be measured. Each of the measures of Practicality measures an idea that was
declared to be an important measure by the decision-maker.
3.3.1.2.1 TECHNOLOGICAL AVAILABILITY
Technology Readiness Levels (TRLs) are described in the Defense Acquisition
Guidebook (Department of Defense, 2004). The nine Technology Readiness Levels (TRL) were
created by the U.S. government to “assess the maturity of evolving technologies.” They allow
the decision-maker to determine the level of development of certain systems during the
acquisition process. TRLs are being used in this case to measure the overall Technological
Availability of the system of systems. The TRL of each component will be averaged to
determine an overall TRL for the entire system. TRLs measure Practicality by determining how
easy it will be to actually construct the system. They are capable of showing whether a
technology is still in the very early development stages or available immediately off the shelf.
This is a Natural-Direct measure since TRLs are widely used within the DoD and are intended to
measure the Technological Availability of systems prior to acquisition. Each of the nine levels is
detailed in Section 3.4. The SV-9 is the most likely location for TRL information. The system
developers have latitude in determining exactly which view will provide the TRLs for each
component. It is possible for the AV-1 to also have a more direct assessment of the TRL of the
entire system.
3.3.1.2.2 ENVIRONMENTAL IMPACT
The ENVIRONMENTAL IMPACT measure determines if the system can be realized within
the environmental constraints of a given location. Since the system is not in any specific location
during the design phase, it is examined prior to construction to determine if it will fit into the
environmental constraints of a given area. For example, to protect the perimeter of an
installation, the base could build a 20-foot high, 10-foot thick concrete wall around the
installation and cover the entire installation with netting to create a protective “bubble,” however,
this method would be cost, time, and environmentally restrictive. Since many environmental
laws would be broken in order to accomplish this feat, it is considered impractical. Most projects
require some sort of Environmental Impact Statement (EIS) before implementation, but at this
level of development, an EIS may not be available. It would be expected that language regarding
the environmental practicality would be included in the AV-1 of a systems architecture, but any
environmental specification being adhered to will be located in the TV-1; therefore, the TV-1 is
the primary location for this measurement. This measure is a Constructed-Proxy measure. This
information is typically included in an EIS, but it is time restrictive for an architecture evaluator
to do this. Environmental Practicality is found by examining the technical standards for
environmental wording. At a later stage of development, this measure will no longer be
Constructed-Proxy, once an EIS is available for review.
3.3.1.2.3 MONETARY PRACTICALITY – INITIAL
In the previous example of perimeter detection, the large concrete wall could not be
constructed due to its impact on the environment. For this measure, the same concept applies,
but to the cost of the wall. The MONETARY PRACTICALITY - INITIAL measure determines if the
program can afford the initial implementation cost of a project. In order for the project to be
completed within the DoD, funds must be realistic and available. The Defense Acquisition
Guidebook includes specific guidelines on funding and affordability (Department of Defense,
2004). There are several places within the architecture where costs may be incorporated. For
this evaluation, the AV-1 will first be examined as it contains overall system information and
more broad concepts such as cost. If information is not found there, other views, such as the
OV-5, which has an optional “cost overlay” function, will be examined. This is a Natural-Direct
measure since dollars are a common unit of measurement and the estimate directly measures the cost
of the project.
3.3.1.2.4 MONETARY PRACTICALITY – MAINTENANCE
This measure examines the “life-cycle cost” of the system. Within the DoD acquisition
system, both initial costs and life-cycle costs are required for a project to reach milestones within
the process (Department of Defense, 2004). Life-cycle costs provide an estimate of how much a
system will cost to maintain throughout the entire effective life of the system, including disposal
at the end of its effective life. While it is difficult to determine monetary constraints in the
future, it is possible to examine future budgets and multi-year plans to see if the gaining office
can work the additional cost into their projected budgets. This measure will also be located
primarily in the AV-1 product. MONETARY PRACTICALITY – MAINTENANCE is also a
NaturalDirect measure for the same reasons as MONETARY PRACTICALITY – INITIAL.
3.3.1.3 Evaluation Measure for Flexibility
Measuring a system’s Flexibility is a difficult concept and requires very strict definition
of the value. Flexibility is defined based on its ability to adapt to changes on the battlefield;
therefore, the measurement seeks to determine its ability to do that. The Flexibility value has
only one measure of effectiveness.
A system’s Flexibility is directly dependent on how easy or possible it is to change the
system configuration quickly and effectively. ADAPTION measures if it is possible to change the
system and the considerations taken to do so. The location of information such as this is not
explicitly stated anywhere in the DoDAF views. The first candidate view for finding this
information is the SV-8, Systems Evolution Description. The SV-8 tells the reader of any
planned future improvements to the system or whether adaptations are possible. This view
should provide information as to how easy it will be to alter the system as well. ADAPTION is a
Constructed-Proxy measure. Since there is no standard system for measuring a system’s
flexibility, a scale must be constructed to measure this concept. Since the value is abstract,
though well defined, no direct measurement will be possible; it is a proxy measurement of things
that would make a system flexible.
3.3.2 Maintainability Measures
In discussions with the decision-maker, it was determined that the Maintainability of a
system is often absent from an architecture. Although there are no DoDAF views that
specifically require explicit information regarding how easy a system is to maintain, the DoDAF
does provide a good vehicle for doing this. The SV-7 product provides the performance
characteristics for the components in the system. Therefore, the evaluator must extrapolate this
information from the SV-7. There are obviously certain parameters to which the system is
designed. Such systems engineering concepts as “Mean Time-Between-Failures (MTBF),”
“System Availability,” “Mean Time-To-Repair,” “Mean Uptime,” “Mean Downtime,” and
“Reliability” have actual equations and methods for evaluation. These things are normally
specified in the project requirements, i.e., the system must be designed to meet certain reliability
standards. For these things to be designed into the system, design calculations must be done at
some point and the standard must be included in the architecture. The SV-7 is the view normally
associated with such performance standards, but these characteristics are typically not included
in the SV-7. The measures for Maintainability are SUPPORTABILITY REQUIREMENTS,
RELIABILITY REQUIREMENTS, SYSTEM REDUNDANCY, and RECOVERABILITY REQUIREMENTS.
3.3.2.1 Evaluation Measure for Supportability
Supportability relates to a system’s ability to be maintained in its operational
environment. The SV-7 must be altered to include this information. The measure for
Supportability is simply named “SUPPORTABILITY REQUIREMENTS.” If the system designers
have incorporated supportability requirements into the design, they should be included in the
SV7. The official measure simply asks, “Have supportability requirements been accounted for?”
The term Supportability Requirement refers to specific systems engineering concepts. Mean
Time-Between-Maintenance (MTBM) and Mean Time-Between-Replacement (MTBR) measure
the time that the system is active between scheduled maintenance. The longer this time is, the
more maintenance is required; therefore, more man-hours are required. The concepts of
(Mean active maintenance time), Mmax (Maximum maintenance time), and (Mean
activepreventative-maintenance-time) measure how much time is actually spent in the
maintenance of the system. These Supportability concepts basically measure the “down-time”
and time between scheduled maintenance. At this point of design, it is important only to
determine if these things have been considered in the design and ensure that they are explicitly
stated in the architecture. At a later time, it will be important to ensure that the times are
acceptable (although the system must meet the basic requirements to be considered a viable
alternative). Through the use of these “career-field standard” equations (Blanchard & Fabrycky,
2006), the concept of
Supportability may be measured, as these are the ideas that are intended to be measured. It can
be assumed that a system is supportable if it exhibits these qualities. The question of
Supportability is a constructed idea for this evaluation, thus, the measure is Constructed-Proxy.
3.3.2.2 Evaluation Measure for Reliability
The Reliability value has characteristics similar to Supportability. Thought it measures a
separate value, the way in which they are measured is similar. The measure “RELIABILITY
REQUIREMENTS” specifically asks, “Have reliability requirements been accounted for?” It also
looks in the SV-7 for similar equations. Reliability seeks to measure the “up-time” of the system
or the length of time that it is operational prior to disruption. This measure assumes that proper
preventive maintenance has been accomplished. The evaluator should look for such equations as
the Availability family (achieved: Aa, inherent: Ai, operational: A0) and MTBF (Blanchard &
Fabrycky, 2006). This measure is also a Constructed-Proxy measure as it uses the same type of
standard equations to measure the concepts involved in this value.
3.3.2.3 Evaluation Measure for Survivability
Survivability departs slightly from the method used to determine the other
Maintainability values. Survivability measures how susceptible a system is to hostile action and
how that action will affect the system. Redundant systems tend to remain operational through
more hostile actions than systems with no redundancies. SYSTEM REDUNDANCY allows a backup
system to take the place of the primary in the case of failure or attack, thereby allowing the
system to remain operational. The “SYSTEM REDUNDANCY” measure determines the degree to
which systems are redundant. The OV-6, Operational Event Trace Description, will provide
some insight as to whether there are intentional system redundancies present. Through the OV-
6’s use of chronological event depiction, it is possible to see which systems are performing
similar actions simultaneously and whether back-up systems exist or not. The measure in this
case is Constructed-Proxy. There is no standard way to measure redundancy; however, looking
in the OV-6, the scorer may be able to get an impression as to whether a system will remain
operational after attack.
3.3.2.4 Evaluation Measure for Recoverability
Recoverability returns to the construct in place for the Supportability and Reliability
values. Recoverability, however, measures “RECOVERABILITY REQUIREMENTS” through the use
of Mean-Time-to-Repair (MTTR) and (Mean active-corrective-maintenance-time). These
ideas measure the length of time that it takes to actually repair the system after some failure.
Whether the failure is through enemy attack or through some other type of system failure, these
equations will account for how long a component takes to repair. The SV-7 is also the source for
these concepts. Recoverability is also a Constructed-Proxy measure, as it uses standard
equations and directly measures the concept of recoverability.
3.3.3 Interoperability Measures
The final branch of System Effectiveness determines how well the system operates with
other systems and between its own nodes. Interoperability was one of the major focuses for the
sponsors of the JFPASS; therefore, its measurement was important to the value of the
architecture. The two Interoperability values are measured through the use of three total
evaluation measures.
3.3.3.1 Evaluation Measure for Interchangeability
Interchangeability deals with the ability of the system components and nodes to be
interchanged between service components. In the Joint environment, being able to operate
smoothly in any service context is vital. The measure “JOINT OPERATIONS” is the sole
Interchangeability measure, and it determines how many services have been involved in the
development of the system. The existing CONOPs for each service must be considered so that
no critical functions are ignored, thereby ensuring a fully interoperable system. The matter of
ensuring that each service component can operate within the context of the new system is a
matter of training, but ensuring that their existing CONOPs and critical operations have been
accounted for will speed the process. The AV-1 document will outline how other services were
incorporated in the design process, but the OV-2, OV-3, and OV-4 may also contain information
regarding the other services’ CONOPs which were incorporated. This measure is a NaturalProxy
type. The number of services is a constructed way of measuring this concept, and its way of
measuring Interoperability is only a proxy for how the system will actually allow the
Interchangeability of different services.
3.3.3.2 Evaluation Measures for Communication
The ability of the system to communicate is essential to its basic operation. Since this is
a largely automated system, the nodes must communicate with one another. In addition, the
system must be able to communicate its processes with the users. The Net-Centric Enterprise
Solutions for Interoperability (NESI) guidance was used to determine the system’s ability to
communicate. The two Communication measures are NESI CONSIDERATION and NESI
EVALUATION.
3.3.3.2.1 NESI CONSIDERATION
“Was NESI guidance taken into account when constructing the architecture?” The
system designers should make use of this detailed guidance for net-centric systems. Since the
JFPASS is required to be net-centric and with the DoD’s shift to net-centric warfare, some
standard of net-centricity is required. Currently, NESI is the only set of consolidated guidance
for determining how a net-centric system should look. At this early stage of development, the
consideration of the NESI guidance in the design is the best way to measure the design
component of Communication. At a later time, it will be possible to measure how net-centric a
system is, but components must be in operation in the field to actually see how they will
communicate. The primary source for NESI DEVELOPMENT is the TV-1. If the NESI
documentation is not listed in the TV-1, it may not have been considered. AV-1 is a possible
back-up as well. NESI DEVELOPMENT is a Constructed-Direct measurement. NESI is becoming
the DoD standard for net-centric designs and if a system is constructed to the specifications
contained within, then it is generally a communicable system.
3.3.3.2.2 NESI EVALUATION
The NESI documentation includes many evaluation measures of its own. This allows the
system developers to perform an initial evaluation to ensure that their system is net-centric.
Through the use of the checklists included in NESI, the NESI EVALUATION measure can
determine how the system developers have done. Again, at this stage of development, it is
important only that the evaluation has been completed. The actual results of the evaluation
should be included as an appendix to the architecture, but the quality of the results will not have
as heavy of an influence until the system reaches a Milestone B approval authority. The NESI
EVALUATION may either be found in the TV-1 or as an appendix to the architecture. This is a
Constructed-Proxy measure. Although NESI is becoming a standard tool, the evaluation is not a
direct measure for this model. It is a way to determine if the evaluation was done through
another system.
3.4 Create Value Functions
Following the creation of the full hierarchy including values and measures, Single
Dimension Value Functions (SDVFs) were assigned to each measure. The SDVFs served to
convert the measure’s score to a value, based on the range of the measure. These SDVFs
converted the individual scales to value units ranging from zero to one. These value scores may
then be summed using the general additive value function. All SDVFs were developed in
coordination with the decision-maker.
3.4.1 Capability Measures Functions
The OPERATIONAL NEEDS measure, under the Purposefulness value was scored on a scale
of 0 to 1. These scores represent the percentage of OPERATIONAL NEEDS addressed by functions.
The SDVF is a monotonically increasing type; therefore, as more of the OPERATIONAL NEEDS of
the system are met, more value is gained. The value is gained in an exponentially increasing
fashion, so that the difference between 0.1 and 0.2 is the same as the difference between 0.7 and
0.8. Figure 3.6 displays the SDVF for OPERATIONAL NEEDS. This SDVF was validated by the
decision-maker on 12 February 2009.
Figure 3.6. OPERATIONAL NEEDS Single Dimension Value Function
The second measure of Purposefulness, THREAT DETECTION, is a binary measure. The
acceptable range of scores for THREAT DETECTION is either “no” or “yes.” This measure
determines if a Threat Detection plan exists. The SDVF for THREAT DETECTION is discrete, with
two bins. All possible value is earned if the score is “yes” and no value is earned if the score is
“no.” Figure 3.7 shows a generic binary SDVF, which may be applied to any binary measure.
All subsequent binary measures, of which there are a total of nine, use the SDVF in Figure 3.7.
The SDVF was validated by the decision-maker on 20 November 2008.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.2
0.4
0.6
0.8
1
Operational Needs
Figure 3.7. Generic Binary Single Dimension Value Function
The last two measures of Purposefulness, THREAT ASSESSMENT and WARNING PLAN, are
also binary measures. These measures determine if either a Threat Assessment or Warning Plan
exists. The SDVF for both is discrete, binary as well. All possible value is earned if the score is
“yes” and no value is earned if the score is “no.” Figure 3.7 shows the binary SDVF, which was
validated for this measure by the decision-maker on 20 November 2008.
The first measure of Practicality, TECHNOLOGICAL AVAILABILITY, is based on a widely
used scale called Technology Readiness Levels. This is a 9-level scale; therefore, the SDVF is
discrete with nine bins. TECHNOLOGICAL AVAILABILITY determines the level at which
technology is available for this project. Table 3.5 shows all TRLs with their definitions. As
TRLs increase, more value is gained. Though this Value Function is discrete, the bins take on an
exponentially increasing value curve. Therefore, the difference between TRL 1 and TRL 2 is
much smaller than the difference between TRL 7 and TRL 8. Figure 3.8 demonstrates this
concept. The SDVF was validated by the decision-maker on 11 January 2009.
No
Yes
Category
0.00
1.00
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Binary SDVF
Table 3.5. Technology Readiness Levels
TRL 1
Basic Principles observed and reported – Lowest level of technology readiness
TRL 2
Technology concept and/or application formulated invention begins
TRL 3
Analytical and experimental critical function and/or characteristic proof of concept
TRL 4
Component and/or breadboard validation in laboratory environment
TRL 5
Component and/or breadboard validation in relevant environment
TRL 6
System/subsystem model or prototype demonstration in a relevant environment
TRL 7
System prototype demonstration in an operational environment
TRL 8
Actual system completed and ‘flight qualified’ through test and demonstration
TRL 9
Actual system ‘flight proven’ through successful mission operations
Figure 3.8. TECHNOLOGICAL AVAILABILITY Single Dimension Value Function
The second measure of Practicality is ENVIRONMENTAL IMPACT. ENVIRONMENTAL
IMPACT is a discrete value function with four bins. It is intended to measure the level of impact
that a given system has on its environment. For this measure, the bins were designed to capture
increasing stringency of environmental laws and restrictions. In this case, the lowest value is that
a system cannot be built within any environmental restrictions, i.e., it will have a vast detrimental
effect on the environment. Contingency Operations environmental constraints are next in level
of restrictiveness. Following that, the CONUS or Contingency constraints provide a higher level
of restriction. Finally, since a system would have to comply with three separate levels of
restriction; CONUS, Contingency, or Host Nation constraints is the most restrictive level of
ENVIRONMENTAL IMPACT. It is assumed that it is easier to design for CONUS constraints than it
is to design for Host Nation constraints, since the corporations and designers are familiar with
the CONUS laws. There is a jump in value between having a system that can be built in
Contingency and CONUS constraints. Figure 3.9 displays the SDVF for ENVIRONMENTAL
IMPACT. This SDVF was validated on 20 November 2009.
Figure 3.9. ENVIRONMENTAL IMPACT Single Dimension Value Function
The third measure of Practicality is MONETARY PRACTICALITY - INITIAL. This measure
is a discrete SDVF with three bins. It is intended to measure the attainment of the ability to
construct a system within budgetary constraints. To this end, the three bins were constructed to
capture situations in which the system estimates fall above, within, and below budget. The
“Within Budget” bin refers specifically to estimates which fall within +/-5% of the given
program budget. Therefore, any estimate falling above 5% of budget qualifies as “Above
Budget,” while any estimate falling below 5% of budget is considered to “Save Money.” Figure
3.10 displays the SDVF for this MONETARY PRACTICALITY measures. It was validated by the
decision-maker on 12 February 2009.
Figure 3.10. MONETARY PRACTICALITY Single Dimension Value Function
The last measure of Practicality, MONETARY PRACTICALITY – MAINTENANCE, is very
similar to the previous measure, except that it refers to the life-cycle cost of the project. The
SDVF converts value in the same way, with the same levels. This is demonstrated in Figure
3.10. This SDVF was also validated on 12 February 2009.
ADAPTION is the single measure of Flexibility. ADAPTION measures the degree to which
the system is able to adapt to changing operational requirements. This is measured on a discrete
scale with five bins. Each bin measures an increasingly easier method of changing the system
configuration. The lowest bins and therefore of lowest value is “Static,” meaning that the system
is not capable of being changed once it is implemented. The next level is “Unacceptable Effort,”
Above
Budget
Within 5% of
Budget
Saves Money
Category
0.00
0.80
1.00
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Monetary Practicality
which refers to a system which can be changed, but is cost and/or time restrictive to actually
make the change to meet the mission at hand. “3rd Party Acceptable Effort” refers to a situation
in which the system can be changed within cost and time constraints, but a 3rd party must be
“imported” to make the change. In this case, users are not capable of changing the system as
needed. The next level is “On-Site Acceptable effort.” In this case, the users are capable of
making the change within cost and time constraints, though it may require such considerations
and consultation from system designers, system downtime, or added cost. The final bin and of
most value to the decision-maker is “Minimal Effort.” This refers to a system that is flexible by
its very nature. Any changes to meet operations requirements are quickly and easily made with
little to no additional time or cost. As the system gets easier to change, more value is added, with
a significant jump in value, 0.4 value units, between “Unacceptable Effort” and “3rd Party
Acceptable Effort.” This value jump represents the value to the decision-maker of having a
system that is capable of being changed. Figure 3.11 shows the SDVF for ADAPTION, which was
validated by the decision-maker on 20 November 2008.
Static
Unacceptab
le Effort
rd Party
3
Acceptable
Effort
On
-
Site
Acceptable
Effort
Minimal
Effort
Category
0.00
0.10
0.50
0.70
1.00
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Adaptation
Figure 3.11. ADAPTION Single Dimension Value Function
3.4.2 Maintainability Measures Value Functions
SUPPORTABILITY REQUIREMENTS is the first of the Maintainability measures.
Specifically, SUPPORTABILITY REQUIREMENTS measures the Supportability value by determining
if the system designers have considered supportability issues. This is a discrete, binary SDVF;
with its range being either “no” or “yes.” If the equations related to supportability were
mentioned, then the system gains full credit (“yes”); if these equations were not considered, the
system earns a “no.” If the SV-7 product does not exist, the system will also score “no,” since
the SV-7 is required to determine the design tolerances used for the component systems.
Figure 3.7 shows the SDVF for SUPPORTABILITY REQUIREMENTS, which was validated on 20
November 2008.
The RELIABILITY REQUIREMENTS measure is similar to Supportability, in that it is also
binary and seeks to measure the use of reliability style equations in the design of the system.
RELIABILITY REQUIREMENTS is the second of the Maintainability values and specifically
measures the Reliability value. Reliability considerations are a different set of design criteria, but
are also found in the SV-7. Value is earned in the same way as SUPPORTABILITY
REQUIREMENTS. Figure 3.7 demonstrates this in the SDVF. This SDVF was validated on 20
November 2008.
SYSTEM REDUNDANCY measures the degree of attainment of the Survivability value. If a
system has redundancies, then it is more likely to survive an attack. To accomplish this
measurement, a discrete value function with four bins was created. As the redundancies on
systems increase, more value is earned. The bins represent a value “jump” when multiple
redundancies are considered for systems. The lowest bin is “No redundancy,” meaning that all
systems are stand-alone and would constitute a loss of mission effectiveness if they were
destroyed. The next bin, “Some Systems have Single Redundancies,” captures the idea that
some systems are given a single back-up to ensure their operation. The decision-maker felt that
it was important for systems to have more than a single redundancy in a force protection
scenario; therefore, the next bin, “Some Systems have Multiple Redundancies,” captures the next
level, which adds a great deal of value to the measure. Finally, “All Systems have Multiple
Redundancies” is the highest bin and level of value. Figure 3.12 displays the SDVF for SYSTEM
REDUNDANCY. This SDVF was validated on 20 November 2008.
Figure 3.12. SYSTEM REDUNDANCY Single Dimension Value Function
No
Redundancy
Some Systems
have Single
Redundancies
Some Systems
have Multiple
Redundancies
All Systems
have Multiple
Redundancies
Category
0.00
0.25
0.75
1.00
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
System Redundancy
The RECOVERABILITY REQUIREMENTS measure is the last of the Maintainability
measures, which specifically measures the value of Recoverability. This measure’s SDVF
follows the same example as both SUPPORTABILITY REQUIREMENTS and RELIABILITY
REQUIREMENTS. It is a discrete, binary SDVF. In this case, the equations generally associated
with recoverability are looked for in the SV-7 product. Figure 3.7 shows the SDVF for
RECOVERABILITY REQUIREMENTS. It was validated on 20 November 2008.
3.4.3 Interoperability Measures Value Functions
JOINT OPERATIONS is the single measure associated with the Interchangeability value.
This measure determines whether multiple service components’ CONOPs have been considered
in the design of the system. This is a binary, discrete SDVF. In order to receive value for this
measure, the system must incorporate all services’ CONOPs. No credit is given unless all four
service components have been considered. The SDVF was constructed in this manner to ensure
that the importance of having all service components is captured. Without all four services, the
interchangeability of the system is of no value. The range for this SDVF is also “yes” or “no.”
The only way to score 100% value or a “yes,” is to have all service components’ CONOPs
considered in the design. Figure 3.7 displays the validated (on 20 November 2008) SDVF.
NESI DEVELOPMENT is the measure created to determine the degree of attainment of the
Communication Value. This measure seeks to determine if the Net-Centric Enterprise Solutions
for Inoperability guidance has been taken into account for the design of the system. This is a
binary, discrete SDVF, measuring either “yes” or “no” as to whether NESI has been used. Figure
3.7 shows the binary SDVF used for NESI DEVELOPMENT. This SDVF was validated on 20
November 2008.
NESI EVALUATION measures the same value in the same way as NESI DEVELOPMENT.
NESI EVALUATION, however, seeks to determine if the system designers have completed an
evaluation on the system. Supplied in the NESI documentation are several checklists and
measures for determining the net-centricity of a system. The system designers must have
completed these evaluations to gain credit for NESI EVALUATION. Figure 3.7 shows the SDVF
for NESI EVALUATION. This SDVF was validated on 20 November 2008.
3.5 Value Hierarchy Weights
To determine the importance of each measure, values must first be individually weighted.
By determining local weights, the global weights are easily found. This study uses primarily the
direct weighting procedure, although an additional procedure using “Tornado Charts” was used
for validation and final weight determination. Table 3.6 shows each System Effectiveness tier’s
values with their global and local weights. The global weights for at each tier sum to 0.6 to
represent the 60% of the fundamental value accounted for by the System Effectiveness branch.
All weights in sections discussed in section 3.5 represent the final weights following validation.
Table 3.6. Value Weights
Value
Tier
Local Weight
Global Weight
System Effectiveness
1
0.6
0.6
Capability
2
0.45
0.27
Maintainability
2
0.275
0.165
Interoperability
2
0.275
0.165
Purposefulness
3
0.6
0.162
Practicality
3
0.3
0.081
Flexibility
3
0.1
0.027
Dependability
3
0.6
0.099
Resiliency
3
0.4
0.066
Interchangeability
3
0.3
0.05
Communication
3
0.7
0.116
Supportability
4
0.35
0.035
Reliability
4
0.65
0.064
Survivability
4
0.6
0.04
Recoverability
4
0.4
0.026
3.5.1 Tier 1 Weights
A top-down approach was used for the majority of the weighting, although validation was
completed in a bottom-up fashion. Tier one was the first tier to be weighted (top-down). With
the split between System Effectiveness and Architecture Quality, relative weights had to be
determined to find how important each of the two branches would be. The local weights were
found to be 0.6 and 0.4, respectively. These weights account for the importance of the
instantiated system versus the architectural products. While the architectural products are very
important, particularly in the acquisition realm, the value of the system being represented is of
more importance.
3.5.2 Tier 2 Weights
The three values considered on Tier 2 of the Hierarchy were Capability, Maintainability,
and Interoperability. A direct weighting scheme (“100 coin method”) was used for these values.
Initially, Capability was weighted at 0.40, with Maintainability and Interoperability both being
0.30 of the value of System Effectiveness. The decision-maker agreed that Maintainability and
Interoperability were in fact of equal weight, but the importance of Maintainability and
Interoperability in relation to Capability was adjusted. The final value of Capability was found
to be 0.45 of System Effectiveness, while Maintainability and Interoperability were 0.275 each.
These final values incorporate the changes made during weighting validation. Capability has the
highest weighting due to the importance that the system is capable of performing its intended
operations. If a system cannot do what it is intended to do, it is little value to the user; therefore,
the 0.45 weighting accounts for this major importance in terms of the system’s ability to do its
job. Maintainability and Interoperability are both important to the decision-maker, but are
overshadowed by Capability.
3.5.3 Tier 3 Weights
Tier three was weighted next using the local weighting method. These values were
examined branch-by-branch to ensure that each value was accounted for and no areas were left
unconsidered. Following the weighting, each value on Tier 3 was validated.
3.5.3.1 Tier 3 Capability Branch Weights
The three values under Capability are Purposefulness, Practicality, and Flexibility. It
was unanimously agreed among the decision-maker and subject matter experts that
Purposefulness was by far the most important value – possibly in the entire hierarchy. With that
knowledge, Practicality and Flexibility were weighted using a swing-weight style approach.
Practicality was agreed to be less important than Purposefulness and Flexibility less important
than Practicality. Next, relative weights were examined. It was determined that Practicality is
approximately three times as important as Flexibility. This decision was made, because if a
system cannot be practically constructed, then its flexibility does not matter. Purposefulness was
then agreed to be twice as important as Practicality. This decision was made because a system
may be practical, but if it does not accomplish its goals, there is no need to construct the system
in the first place. This process yielded weights of 0.60 for Purposefulness, 0.30 for Practicality,
and 0.10 for Flexibility.
3.5.3.2 Tier 3 Maintainability Branch Weights
The Maintainability branch is the only branch within System Effectiveness that contains a
fourth tier of decomposition. The first step for weighting this branch was to determine the
weights of the tier-three values. The tier-four values were examined after all tier-three values
were weighted. Maintainability has sub-values of Dependability and Resiliency. Dependability
refers to a system’s maintainability during peace-time operations, and Resiliency refers to its
maintainability during hostile actions. The weighting for these values was based on the
frequency of occurrence. Since the U.S. military has more assets that are operating in peace-time
operations, Dependability was determined to be more important. While the military has more
operations engaged in peaceful actions, the operations vulnerable to hostile actions are
considered to be of more importance to the completion of the National Security and National
Military strategies. This idea was confirmed by Subject Matter Experts. Therefore, Resiliency
was weighted at 0.40 and Dependability was weighted at 0.60. Dependability being only slightly
more important accounts for the mission impact of contingency operations, but the overall
importance of the military’s peace-time operations.
3.5.3.3 Tier 3 Interoperability Branch Weights
The Interoperability branch was the last branch of System Effectiveness to be weighted.
This branch accounts for Interchangeability and Communication. The swing weighting method
was used for this weighting. It was determined first the Communication was more important to
the decision-maker; meaning that the nodes communicating effectively is more important than
the components having the ability to be interchanged. With Interchangeability being the less
important value, the discussion led to a determination that Communication was four times more
important than Interchangeability. This order of magnitude captures the extreme importance of
the initial nodes performing their communication function first. Their Interchangeability falls
behind, since without communication, there would be no need to interchange.
3.5.4 Tier 4 Weights
Following the completion of tier-three weighting, tier-four weights were examined. Only
three values of the hierarchy have tier-four values. These values were examined individually to
find their local weights. Under the Maintainability branch, the values of Dependability and
Resiliency were decomposed by one additional level, giving four tier-four values under
Maintainability.
Dependability was examined first. The sub-values of Supportability and Reliability
compose the idea of peace-time dependability. Supportability represents the ability to maintain
the system and Reliability represents the system’s ability to continue standard operations if
maintained properly. Supportability was found to be less important than Reliability, based on the
need for system “up-time.” If a system is difficult to maintain, the system will be offline more
often. The most important thing about standard peace-time operations is that the system is
operational more often than it is not. Therefore, the decision-maker was willing to sacrifice
maintainability for more operational time. Initially, Reliability was found to be twice as
important as Supportability, but the validation phase found that Reliability was actually 0.65 of
the value of Dependability and Supportability was 0.35 of the value. This determination was
made using the “100 coin method” during validation.
The second aspect of Maintainability, Resiliency, was weighted next. A similar
consideration was made for this value. It is of more importance that the system remains
operational during an attack than it is easily repaired after the attack. Generally, more time is
available following an attack and repair is not as critical; therefore, ensuring that the system
never goes down is important. The direct weighting method was used for the values of
Survivability and Recoverability. Survivability was found to be 0.60 of the value of Resiliency
and Recoverability 0.40.
3.5.5 Weight Validation
Following the initial value weighting, the entire Hierarchy was reviewed to ensure
weights were accurate. The weights were validated using the “tornado chart” weighting method.
Each level of the hierarchy was examined from a “bottom-up” method. The bottom-up method
was chosen, so that as measures were re-weighted and adjusted, the weights of the higher levels
could immediately reflect the changes. A major advantage of the tornado weighting method is
that it allows the decision-maker to see the relative importance of values in different branches
and ensure that their relative placement is correct; therefore, global weights were used during the
validation process. Global weights are the weights of the value relative to all other values on the
same tier of the hierarchy.
3.5.5.1 Tier 3 Validation
Tier three was the first level to be validated using a “stacked” tornado chart. Since there
is not a full fourth tier, it was not possible to start on the fourth level. By stacking Tier 4 values
and displaying on a single bar chart, the decision-maker was able to see both the tier-three values
and the tier-four values at the same time. Weighting was first done using separate Tier 1
branches (System Effectiveness and Architecture Quality) and then combined into a single chart.
In each chart, the relative placement of each value was individually examined to ensure its
placement in the overall hierarchy. Figure 3.13 shows the tier-three global weights for each
value in the system branch. When local weights are multiplied up through the hierarchy to be
converted to global weights, their order of importance is found.
Figure 3.13. Tier 3 System Effectiveness Global Weights Stacked
In the initial presentation of the value weights, Purposefulness was weighted similarly,
but shared the Capability value with only one other value. Once Flexibility was moved to the
Capability branch, each of the three values under Capability were re-weighted. Purposefulness
maintained its original weight of 0.60 locally, giving it a global value of 0.162. It was
unanimously agreed that Purposefulness should be the most important value globally.
Practicality was discussed next. Its original weight was 0.40, but with the addition of Flexibility,
was re-weighted to be 0.30 of the value of Capability and 0.081 global value. Practicality was
important enough to be in the top of values of importance, therefore its weight was monitored to
ensure that it remained within the top five values. Its final global importance was fifth overall.
Flexibility was weighted at 0.10 as discussed in section 3.5.3.1. Its global value was 0.027.
Flexibility’s final importance was fourteenth of importance globally.
7
0.02
0.050
0.
040
081
0.
0.064
6
1
0.1
162
0.
0.02
6
0.
035
0.000
0.020
0.040
0.060
0.080
0.100
0.120
0.140
0.160
0.180
Flexibility
Interchangeability
Resilliancy
Practicality
Dependability
Communication
Purposefulness
ility
upportab
ty and S
Reliabili
overabili
and Rec
vability
Survi
ty
The tier three Maintainability values were examined next. Dependability and Resiliency,
with their sub values of Reliability, Supportability, Survivability, and Recoverability were
validated by adjusting the weight of the subvalues to determine what effect it had on the overall
values of Dependability and Resiliency. Reliability and Supportability, the Dependability
subvalues on tier four, were originally weighted at 0.60 and 0.40 locally. These weights were
adjusted to 0.65 for Reliability and 0.45 for Supportability. This change was made to account for
the fact that Reliability, being the system’s “up-time” is more important than the ease of
maintenance for a system. The 5% change to each value was made to adjust their standings in
the tornado chart. When examined together, these values make up 100% of the Dependability
value. Their global values were equal to 0.064 for Reliability and 0.035 for Supportability.
These global values were used to examine the value’s standings on the Tornado chart. This
change placed Dependability above Understandability in the global rankings Tornado chart
(Figure 3.46) and in the top five of all values. Dependability is more important than
understandability, because the system’s ability to continue performance (maintenance time and
system up-time) is more important to the decision-maker than the ease of reading architectural
products. It also assured Dependability’s place above Practicality on the System Effectiveness
values only tornado chart (Figure 3.44). Dependability itself (Reliability plus Supportability) is
weighted as 0.60 local and 0.099 global on tier three. Dependability is the third most important
value globally.
Resiliency is composed of the subvalues Survivability and Recoverability. Survivability
and Recoverability were both accepted by the decision-maker with the weights as presented. The
Survivability value was presented as 0.60 local and 0.04 global. Recoverability was 0.40 local
and 0.026 global. These weights were assigned as such to account for the importance of the
system remaining operational during an attack. It is important for the system to be recovered
quickly, but the more critical time period is during the attack itself. These subvalues combine to
form the tier three value of Resiliency. Resiliency, on tier three is weighted as 0.40 local and
0.066 global. These weightings place Resiliency as the sixth most important value globally.
Finally, the Interoperability branch was examined on tier three. The two tier three
Interoperability values were Interchangeability and Communication. Due to the movement of
Flexibility from the Interoperability branch to the Capability branch, Communication was
reweighted to 0.70 from 0.60 locally. This left Communication at 0.116 global value, making it
the second most important value globally. Interchangeability remained at 0.30 local weight. Its
global value was set at 0.05 and was the seventh most important global value.
Figure 3.14 shows all global values on tier three of the hierarchy. This Tornado chart
includes both System Effectiveness values and Architecture Quality values. As shown on this
chart, six of the seven most important values are from the System Effectiveness branch of the
hierarchy. These rankings were validated based on the importance of the system to perform as
expected. The Architectural products are also important, but they only represent the instantiated
system and therefore rank lower globally. The one Architecture Quality value in the top seven is
Understandability. This value is placed in its location due to the importance of the architectural
products to be understood and to effectively communicate the concepts of the future system to a
wide audience.
Purposefulness
Communication
Dependability
Understandability
0.
162
nd Supporta
Practicality
Resilliancy
Interchangeability
Longevity
Protectability
Controllability
Subscribability
Compliancy
SME Input
Flexibility
Tailorability
Scalability
Traceability
Consistency
Evolvability
0.000
nd Simplicit
bility
y
0.1
16
Reliability a
Readability a
coverability
0.064
0.
035
0.059
0.0
39
bility and Re
0.
081
Surviva
0.
040
0.026
0.050
0.
042
0.
040
0.030
0.030
0.030
0.030
0.027
Archite
cture
Quality m
Effectivene
ss
0.024
0.024
Syste
0.020
0.020
0.012
0.020 0.040 0.060
0.080 0.100 0.120 0.140 0.160
Figure 3.14. Tier 3 All Global Values
3.5.5.2 Tier 2 Validation
Tier two validation was done using the same method as tier three. Figure 3.46 shows the
final global and local weights for the System Effectiveness values. Following the weighting of
the tier three values, tier two values were simply checked for their placement among other
System Effectiveness values and other tier 2 values. Figure 3.15 shows all values in Tier 2.
Capability was validated at 0.45 local weight and 0.27 global weight. Interoperability and
Maintainability were both validated at 0.275 local and 0.165 global weight.
Figure 3.15. Tier 2 System Effectiveness Local and Global Weights
When examined among other tier 2 values, Capability was still the most important value
with its 0.27 global value. Interoperability and Maintainability were the next most important tier
two values at 0.165 global weight each. This ranking confirmed Capability as the highest
weighted value.
3.5.5.3 Tier 1 Validation
The final validation completed was for the tier one values. The only tier one values are
the two major branches of the hierarchy. System Effectiveness was weighted at 0.60 of the
fundamental objective and Architecture Quality at 0.40 of the fundamental objective. This
0.
275
0.
275
0.450
0.1
65
0.1
65
0.2
70
0.000
0.050
0.100
0.150
0.200
0.250
0.300
0.350
0.400
0.450
Maintainability
Interoperability
Capability
Global Weight
Local Weight
accounts for the higher importance of the instantiated system, versus the architectural products
that represent the system.
Chapter 4. Analysis and Results
Following the finalization of the value hierarchy the existing architecture was scored and
future alternatives were generated. For the Value-Driven Enterprise Architecture, the JFPASS
system was evaluated first. Following this baseline score, the other generated alternatives were
scored for comparison purposes. A sensitivity analysis was then completed to determine the
effect that the weights of each value had on the final score as well as the impact of each measure
to the final score.
4.1 JFPASS Architecture Scoring
The JFPASS system was examined first. Each measure in the hierarchy was examined
individually and assigned a score based on the existing architecture. Since the existing
architecture is at a very early stage of development, there are portions of value that have not yet
been earned. The instantiated system does not yet exist; therefore, only the architectural products
were used as data sources in the scoring process. Some measures may score higher based on
information not yet included in architectural products, but since this information is not readily
available to a third party reviewer and is therefore not verifiable, it cannot be included in the
scoring of the system. Some areas of value are located in very specific architectural products;
therefore, if these products do not yet exist, some measures cannot be accurately scored until
those products are created. Value will be earned incrementally as new information is available
during the life cycle of the project. The scoring presented here is the baseline for future
improvements. The views available for scoring the JFPASS are listed in Table 4.1. Table 4.2
shows the final scores for each of the measures. To score the alternatives, the hierarchy was
entered into a proprietary software package, Hierarchy Builder v1.01© (Weir, 2006). The
Hierarchy Builder © package allows for all aspects of the scoring and analysis of a hierarchy.
Hierarchy Builder © also generates the Single Dimension Value Functions (SDVF) as entered by
the user. To determine the equations associated with the continuous SDVFs, a trendline was fit
to the curve and the equation was exported from Microsoft Excel©.
Table 4.1 Available JFPASS views
AV-1
SV-1
OV-1
SV-2
OV-2
SV-4
OV-4
SV-6
OV-5
TV-1
OV-6c
Table 4.2 Scores and Associated Values
Alternative Name
JFPASS 17 February 2009
Measure
Score
Global
Weight
Value
OPERATIONAL NEEDS
0
0.041
0.001
THREAT DETECTION
Yes
0.041
1.000
THREAT ASSESSMENT
Yes
0.041
1.000
WARNING PLAN
Yes
0.041
1.000
TECHNOLOGICAL AVAILABILITY
TRL 1
0.02
0.050
ENVIRONMENTAL IMPACT
CONU.S. and Contingency
constraints
0.02
0.800
MONETARY PRACTICALITY –
INITIAL
Cost Unknown
0.02
0.000
MONETARY PRACTICALITY –
MAINTENANCE
Cost unknown
0.02
0.000
ADAPTATION
Static
0.027
0.000
SUPPORTABILITY REQUIREMENTS
No
0.035
0.000
RELIABILITY REQUIREMENTS
No
0.064
0.000
REDUNDANCY
Some systems, single redundancy
0.04
0.250
RECOVERABILITY REQUIREMENTS
No
0.026
0.000
JOINT OPERATIONS
Yes
0.033
1.000
NESI DEVELOPMENT
Yes
0.066
1.000
NESI EVALUATION
No
0.066
0.000
4.1.1 Capability Measures Scoring
OPERATIONAL NEEDS primarily requires the use of the AV-1 and SV-5, although
information may also be found in the OV-1, OV-3, OV-5, and SV-7. The AV-1 provides a list of
OPERATIONAL NEEDS upon which to base the analysis. At a minimum, the AV-1 provides the
system purpose and goals. The SV-5 is then used to trace those requirements down to functions
or components, thereby ensuring that each Operational Need is met by some portion of the
system. The AV-1 provided by the sponsoring organization had no information regarding the
operational needs. In addition, there was no SV-5 from which to trace functions to components.
Therefore, the OPERATIONAL NEEDS measure scores 0% in this evaluation with zero associated
value.
THREAT DETECTION was scored based on the existence of a Threat Detection plan. At a
later point in the development of the project, the Threat Detection plan itself may be graded, but
at this point, its existence was the only important value. The OV-5 was examined in this case to
determine its existence. OV-1 and OV-3 were also used for back up and additional information
for the Threat Detection plan scoring. The OV-5 has an operational activity devoted to the
DAWDR concept of Detect. Under this activity are many sub activities outlining exactly how
the JFPASS will accomplish the Detect activity. In addition to the information in the OV-5, the
OV-1 shows a system component devoted to threat detection as well. The THREAT DETECTION
measure scored “Yes” with an associated value of one.
The same architectural views were used in the evaluation of THREAT ASSESSMENT as
were used for THREAT DETECTION. The OV-5 included a similar activity called “Assess,” which
accomplished the activities required in a Threat Assessment Plan. The OV-1 also includes an
assessment component in the system scope. THREAT ASSESSMENT scored “yes” in the evaluation
with an associated value of one.
The OV-5 and OV-1 were again employed in the scoring of WARNING PLAN. The OV-5
includes a Warn activity. The OV-1 also includes a “Wide Area Alert” component. There are
some other aspects of the OV-1 which allude to a Warning plan, including the “Chemical
Sensors,” “Lan,” and “C2,” all of which may accomplish a portion of the warning plan.
WARNING PLAN scored “Yes” in the evaluation with an associated value of one.
The SV-8 view typically includes information related to the Flexibility of a system. Since
an SV-8 does not yet exist for the JFPASS, other views were examined for information regarding
ADAPTION. No information was found to score ADAPTION in the existing architecture products.
ADAPTION was scored as “static” in this evaluation. A flexible system must include information
regarding flexibility, as well as how the system may be altered in the architecture. A “Static”
score yields zero value in this hierarchy.
The Technology Readiness Level (TRL) Scale was employed for the TECHNOLOGICAL
AVAILABILITY measure. Typically, an SV-9 would include the necessary information to
determine the TRL of a component. Since there was no SV-9 product, specific information on
system components was searched for among the existing products. The TRL was not explicitly
or implicitly stated in any of the existing products nor was there information to extrapolate the
TRL for any component. Therefore, TECHNOLOGICAL AVAILABILITY was scored as “TRL 1 or
No data” with an associated value of 0.05. The lowest level of TRL was assigned since this level
accounts for components that have not yet entered any phase of development.
The ENVIRONMENTAL IMPACT measure was scored using the TV-1 product. Section 3.93
of the TV-1 is the Environmental section. The guidance documents listed here include Mil
Standards for Environmental Engineering, Electromagnetic Compatibility, and Interference
Standard requirements. The documents included represent a cross section of some of the types
of environmental guidance documents that must be considered in the design of an
environmentally practical system. The inclusion of these documents in the TV-1 assumes the
inclusion of these documents in the design of the system. The standards being used in this
system are military and United States federal standards; therefore, the system was scored
“CONUS and Contingency constraints” with an associated value of 0.8. The inclusion of
military standards shows that contingency environments have been considered.
Indicators for the MONETARY PRACTICALITY – INITIAL measure may be found in the OV5
product, although there are no specifically required views in DoDAF for estimates. Cost may be
included in a layer of the OV-5 for some tools. If no cost is included in the OV-5, the OV-7 is
also examined for possible cost elements. The JFPASS architecture had no cost information in
any of the available views; therefore, the MONETARY PRACTICALITY measure scored “Cost
Unknown” with an associated value of zero.
MONETARY PRACTICALITY – MAINTENANCE measure also scored “Cost Unknown” in
this evaluation. The same views were examined for costs elements, but JFPASS did not include
any cost information. The life-cycle cost for the system is an important consideration for
decision-makers, but this estimate was not available in the provided views, so the value
associated with MONETARY PRACTICALITY – MAINTENANCE was zero.
4.1.2 Maintainability Measures Scoring Score
Every system must be designed with tolerances for components to ensure that the system
is within design parameters. DoDAF includes a vehicle for these requirements by way of the
SV-7 product. Since the JFPASS does not currently include an SV-7; therefore, the existence of
design standards for supportability cannot be verified. None of the other provided products
include the information either. SUPPORTABILITY REQUIREMENTS therefore scores “No” with an
associated value of zero.
Reliability Requirement considerations must also be taken into account in the design of a
system. Since the SV-7 does not yet exist in the JFPASS, the equations and values required to
ensure the inclusion of Reliability could not be located. The architecture must include some
verification of these items to ensure that they are considered in the system design. RELIABILITY
REQUIREMENTS scored “No” with an associated value of zero in this evaluation.
It is assumed that if redundancy exists of critical systems, these systems will be more
survivable during an attack. SYSTEM REDUNDANCY ensures more system up-time during hostile
action. The OV-6 event trace shows some evidence of system redundancies. The JFPASS
includes one OV-6c for a single system activity. This view shows some evidence of redundancy
in this activity. Additional views show some evidence of redundancy as well. Based on the data
provided in these views, SYSTEM REDUNDANCY was scored as “Some Systems, Single
Redundancy.” This gave the SYSTEM REDUNDANCY measure a value of 0.25.
The lack of an SV-7 product does not enable the verification of RECOVERABILITY
REQUIREMENTS. These equations must be verified to ensure their inclusion in the architectural
design. Since they cannot be verified, this measure scored “No” with an associated value of
zero.
4.1.3 Interoperability Measures Scoring
The JOINT OPERATIONS measure is based on the consideration of all service components.
Since it is a critical requirement that no service be left out, this measure is scored either “yes” or
“no.” In this instance, JOINT OPERATIONS scored “yes” based on information drawn from a
variety of views. The AV-1, OV-2, OV-3, OV-4, and SV-2 were the primary resources used for
this evaluation. The AV-1 specifically contains references to all service documentation. In the
available Operational Views, there is evidence of other service requirements, such as port
security and convoy security which alludes to specific service requirements and operations. The
associated value with this measure is one.
The TV-1 is the primary view employed in the evaluation of the NESI DEVELOPMENT
measure. Section 3.1 and 3.2 of the TV-1 refer to Information Technology. Specifically, Section
3.1.2, Common Infrastructure Data Format Standards, and section 3.1.3, Network Management,
relate to the concept of net-centricity. There are also documents related to wireless
communications. While the Net-Centric Standards for Interoperability (NESI) is not referenced
specifically in the TV-1, the concepts set forth in NESI are accounted for based on the documents
provided in the TV-1. NESI DEVELOPMENT scored “Yes” with an associated value of one in this
evaluation.
The evaluation criterion and checklists provided in the NESI Guidance were not
completed for this system. There is no evidence in the provided architecture that the system was
evaluated for compliance with NESI. NESI EVALUATION scored “No” in this evaluation with an
associated value of zero.
4.2 Deterministic Analysis
With all measures scored, a deterministic analysis was performed to find a single,
aggregate score for the entire project. Using the general additive value function, this score
combines the associated values for each score as well as the weight of each measure. This score
is simply a weighted average of these numbers. The Hierarchy Builder© software was used to
generate graphs and do a comparative analysis of possible alternatives.
4.2.1 Additive Value Function
The general additive value function was used to create the JFPASS final score. The
general additive value function is represented as (Kirkwood, 1997, p. 230):
Eq 4.2
Where v(x) = the overall score of the alternative, = the value of the score on the ith measure,
= weight of the ith measure, n = the total number of measure, = 1.0
Table 4.3 shows all measures and weights with the Architecture Quality measures included.
To obtain a score for the full system, the measures for Architecture Quality must also be used. It
is possible to examine the two branches separately, but this will give the decisionmaker an
incomplete picture of the entire system. The deterministic analysis for the System Effectiveness
branch of JFPASS was completed using the entire system as well as the System
Effectiveness branch alone. For all analyses, the Architecture Quality score was held static from
Cotton and Haase’s (2009) evaluation of the system.
Table 4.3 All Measures and Weights
Measure Description
Global Weight (
)
Value
OPERATIONAL NEEDS
0.041
0.000
THREAT DETECTION
0.041
0.041
THREAT ASSESSMENT
0.041
0.041
WARNING PLAN
0.041
0.041
TECHNOLOGICAL AVAILABILITY
0.020
0.001
ENVIRONMENTAL IMPACT
0.020
0.016
MONETARY PRACTICALITY - INITIAL
0.020
0.000
MONETARY PRACTICALITY - MAINTENANCE
0.020
0.000
ADAPTATION
0.027
0.000
SUPPORTABILITY REQUIREMENTS
0.035
0.000
RELIABILITY REQUIREMENTS
0.064
0.000
SYSTEM REDUNDANCY
0.040
0.010
RECOVERABILITY REQUIREMENTS
0.026
0.000
JOINT OPERATIONS
0.033
0.033
NESI DEVELOPMENT
0.066
0.066
NESI EVALUATION
0.066
0.000
ACCESS
0.022
0.022
PRODUCT LOCATABILITY
0.011
0.011
DOCUMENT PROTECTION
0.033
0.033
PROTECTION
0.033
0.011
FILE MANAGEMENT
0.021
0.000
FILE FORMAT
0.021
0.021
CONNECTIONS
0.013
0.008
ARCHITECTURE REDUNDANCY
0.013
0.013
ARCHITECTURE ECONOMY
0.013
0.013
OV READABILITY
0.029
0.027
SV READABILITY
0.029
0.021
SCALE
0.024
0.014
DECOMPOSITION
0.024
0.024
TOOL FORMAT
0.012
0.012
DODAF COMPLIANCY
0.025
0.025
REQUIREMENT TRACEABILITY
0.025
0.000
INTERNAL CONSISTENCY
0.013
0.009
EXTERNAL CONSISTENCY
0.013
0.009
SME INVOLVEMENT
0.013
0.000
SME EFFECTIVENESS
0.013
0.000
4.2.2 JFPASS Analysis
The final score for the total system was found to be 0.538 out of 1.000. Figure 4.1 shows
a stacked bar chart of each of the two major Tier 1 values. The System Effectiveness branch,
though weighted higher (0.6) in the fundamental objective, received less of the overall weight.
JFPASS 17 February 2009 0.538JFPASS Current System
System Effectiveness Architecture Quality
Figure 4.1 JFPASS Score Fundamental Objective - Values
Figure 4.2 shows the VDEA Score score sorted by measure. In this example, each
measure’s value and global weight accounts for a portion of the bar chart. While Purposefulness
is the highest weighted value, it has four measures intended to measure the concept. Therefore,
each of those four measures have a smaller global weight. NESI DEVELOPMENT and NESI
EVALUATION, however, have only one measure to determine the degree of attainment of the
value; therefore, their measures’ global weights are higher. The high global weight and
associated value of NESI DEVELOPMENT and NESI EVALUATION cause these measures to be
among the highest weighted measures. It is evident in Figure 4.2 that a large portion of value
was lost due to the lack of a performing a NESI EVALUATION and the lack of OPERATIONAL
NEEDS attainment. Three of th four Maintainability measures rank third, sixth, and seventh in
possible global weight of the system. Some value was earned by SYSTEM REDUNDANCY, but no
value was earned by Reliability or SUPPORTABILITY REQUIREMENTS.
Of the 0.60 total possible value of the fundamental objective, the System Effectiveness
branch of the hierarchy earned 41.3% of its value. This equates to 0.248 of the 0.600 possible
value units for System Effectiveness. Figure 4.3 shows the measures under the System
Effectiveness branch only. This figure and score does not take into account any of the
Architecture Quality measures or scores.
JFPASS Current SystemJFPASS 17 February 2009 0.538
NESI DevelopmentNESI Evaluation
Reliability RequirementsOperational Needs
Threat DetectionThreat Assessment
Warning PlanRedundancy
Supportability RequirementsAccess Control
Document ProtectionJoint Operations
DoDAF CompliancyOV Readability SV
ReadabilityAdaptation
Recoverability RequirementsScale
DecompositionAccess
File ManagementFile Format
Technological AvailabilityEnvironmental Impact
Monetary Practicality - InitialMonetary Practicality -Maintenance
Requirement TraceabilitySME Effectiveness
SME InvolvementConnections
Architecture RedundancyArchitecture Economy
Tool FormatProduct Locatability
Internal ConsistencyExternal Consistency
Figure 4.2 JFPASS Score Fundamental Objective - Measures
NESI Development
Reliability
Requirements
Threat Detection
Warning Plan
Supportability
Requirements
Adaptation
Technological Availability
Monetary Practicality -
Initial NESI Evaluation
Operational Needs
Threat Assessment
Redundancy
Joint Operations
Recoverability Requirements
Environmental Impact Monetary
Practicality - Maintenance
JFPASS 17 February 2009 0.413
JFPASS Current System
Figure 4.3 System Effectiveness Score – Measures
When examining the System Effectiveness branch alone in figure 4.3, it is evident where
value was lost in the system. NESI EVALUATION, RELIABILITY REQUIREMENTS, and
OPERATIONAL NEEDS are the three highest ranking measures that did not score any value.
SYSTEM REDUNDANCY earned 0.25 of its total possible value, but none of the other
Maintainability measures, SUPPORTABILITY and RECOVERABILITY REQUIREMENTS, earned any
value. JOINT OPERATIONS earned full value, but ADAPTATION, TECHNOLOGICAL AVAILABILITY,
MONETARY PRACTICALITY – INITIAL, and MONETARY PRACTICALITY – MAINTENANCE did not
earn any value.
The next area of the hierarchy to be analyzed was the Capability branch of the System
Effectiveness branch. Capability earned 51.4% of its total possible value (45% of System
Effectiveness). The measures providing value to this branch are THREAT DETECTION, THREAT
ASSESSMENT, WARNING PLAN, TECHNOLOGICAL AVAILABILITY, and Environmental
Practicality. Figure 4.4 shows the bar chart of this branch. The Interoperability branch and
Maintainability branch each only contributed one measure worth of value to the total score.
Additional bar charts for the System Effectiveness Branch are found in Appendix C.
JFPASS 17 February 2009 0.514
Operational NeedsThreat Detection Threat
AssessmentWarning Plan
AdaptationTechnological Availability
Environmental ImpactMonetary Practicality - Initial
Monetary Practicality - Maintenance
Figure 4.4 Capability Score – Measures
4.2.3 Gap Analysis
The actual generation of alternatives was completed via a gap analysis. This gap analysis
allows the model builder to determine how much of an affect each measure has on the overall
score. By adjusting the score of each measure and evaluating the total system score, the gap
between the current value and highest possible (or lowest possible) value can be found. Table 4.4
shows the results of this sensitivity analysis in the range of each measure’s scoring possibilities.
The range is shown in the low and high columns. The “current” column shows the current
overall system score, with the final column showing the gap between the current system score
and the highest possible score that could be attained by maximizing the measure in question.
Table 4.4 Gap Analysis Score Ranges
Global Value
Measure
Low
High
Current
Gap
OPERATIONAL NEEDS
0.538
0.578
0.538
0.040
THREAT DETECTION
0.497
0.538
0.538
0.041
THREAT ASSESSMENT
0.497
0.538
0.538
0.041
WARNING
0.497
0.538
0.538
0.041
TECHNOLOGICAL AVAILABILITY
0.538
0.557
0.538
0.019
ENVIRONMENTAL IMPACT
0.521
0.542
0.538
0.021
MONETARY PRACTICALITY -
INITIAL
0.538
0.558
0.538
0.020
MONETARY PRACTICALITY -
MAINTENANCE
0.538
0.558
0.538
0.020
ADAPTATION
0.538
0.565
0.538
0.027
JOINT OPERATIONS
0.505
0.538
0.538
0.033
NESI DEVELOPMENT
0.472
0.538
0.538
0.066
NESI EVALUATION
0.538
0.604
0.538
0.066
SUPPORTABILITY
REQUIREMENTS
0.538
0.572
0.538
0.034
RELIABILITY REQUIREMENTS
0.538
0.602
0.538
0.064
SYSTEM REDUNDANCY
0.528
0.567
0.538
0.039
RECOVERABILITY
REQUIREMENTS
0.538
0.564
0.538
0.026
4.2.3.1 Capability Measurement Sensitivity
The OPERATIONAL NEEDS measurment was altered by increasing the score by 10% or 0.1
incrementally. Each incremental increase created a new alternative. The ten created alternatives
showed a gap of 0.04 value units between the lowest score of OPERATIONAL NEEDS and the
highest score.
The THREAT DETECTION measure is a binary measure; therefore, there are only two
scoring possibilities. Since the current score is set to “yes,” the alternative to this measurement
scenario is if the measurement was scored “no.” The alternative was worth less value than the
current situation, with a loss of 0.041 value units. THREAT ASSESSMENT is the same in all ways to
the THREAT DETECTION measurement in sensitivity. The range is also 0.041, as well as the binary
nature. The WARNING PLAN measure is also similar to THREAT DETECTION and THREAT
ASSESSMENT. It also has a range of 0.041 meaning that if it were to be scored “no,” the
maximum global system value lost would be 0.041.
TECHNOLOGICAL AVAILABILITY had a total of nine generated alternatives for sensitivity
analysis. The TECHNOLOGICAL AVAILABILITY measurement was set to each TRL level for each
alternative and the range for the TECHNOLOGICAL AVAILABILITY measure was found. The range
was 0.019 change in value unit from the lowest score to the highest score of TECHNOLOGICAL
AVAILABILITY. The current score of this measure was “Unknonwn/TRL 1,” therefore any
additional information or change to this measure would cause an increase to the total system
score.
ENVIRONMENTAL IMPACT measurement has four possible score settings. This gives
ENVIRONMENTAL IMPACT a range of 0.021, from 0.521 to 0.538 total system scores. The current
system score is 0.538 with only one higher score possible. Therefore, only 0.004 value units are
available for ENVIRONMENTAL IMPACT.
MONETARY PRACTICALITY – INITIAL has four possible score categories. The current score
of this measure earns no value, therefore any change to the score would cause an increase to the
total system score. The possible change is 0.20 globally. Figure 4.32 and 4.33 show the global
and local measure sensitivity. MONETARY PRACTICALITY – MAINTENANCE has the same range
and numerical value possibilities as MONETARY PRACTICALITY – INITIAL.
ADAPTION has five possible value categories. Currently the ADAPTION measure earns no
value, therefore any increase in measurement will cause an increase in total system score. The
range of scores for ADAPTION is 0.027 globally. While ADAPTION has a smaller effect on the total
system score, any value earned helps the overall system score.
4.2.3.2 Maintainability Measurement Sensitivity
SUPPORTABILITY REQUIREMENTS is a binary measure, which is currently measured at its
lower value possibility. If the score is set to “yes,” the possible change in value is 0.034 value
units. This change in overall score indicates that this measurement will have a significant impact
on the overall score.
The RELIABILITY REQUIREMENTS measure is also a binary scoring set. Since the
Reliability value has a higher global weight, the range in score change is larger for this measure of
Maintainability. The VDEA score range is 0.064 for RELIABILITY REQUIREMENTS.
RECOVERABILITY REQUIREMENTS is also a binary measure. It is similar to both
SUPPORTABILITY REQUIREMENTS and RELIABILITY REQUIREMENTS and is currently scored as
“no.” The range for this measure is 0.538 – 0.564, meaning that the highest possible value
change between the lower score and higher score is 0.026.
The SYSTEM REDUNDANCY measure is rated on a scale of four possible levels. The
current system is scored at the second level. The range between the lowest and highest scoring
possibilities is 0.039 value units. As one of the four Maintainability measures, SYSTEM
REDUNDANCY is an important design consideration, which is evident in its effect on the total
system score.
4.2.3.3 Interoperability Measures Sensitivity
The JOINT OPERATIONS measurement scored “yes” in this evaluation, giving it its full
possible value. If it had been scored no, the greatest change to the total system score would have
been 0.033 value units, taking the system score to 0.505. This is a relatively large change for a
single measure, indicating that the JOINT OPERATIONS measurement is of great importance.
The NESI DEVELOPMENT measurement is a binary measure, currently scored at its higher score
of “yes.” NESI DEVELOPMENT is one of the more important measures in this evaluation, with a
global weight of 0.066. Therefore, its impact on the total score is higher than most other
measures. The swing between its higher and lower score possibilities is 0.066 value units. This
makes the lower possible range of the VDEA score to 0.472 for this measure.
The final measure on the System Effectiveness branch of the hierarchy was NESI
EVALUATION. This measure also has a high global weight making it a relatively important
measure. In this case, the system scored at the lower possibility, therefore if the score was
changed to “yes,” the change in VDEA score would be 0.066, taking the total score to 0.604.
4.2.4 Alternative Generation
Step six of the Value-Focused Thinking (VFT) process is the Alternative Generation step.
In the standard VFT process, at this point, the hierarchy would be used to find alternatives which
fit the evaluation criteria. In the case of the Joint Force Protection Advanced Security System
(JFPASS), though, a single alternative was created by the decision-maker. However, several
additional alternatives were generated as comparison criteria for the decision-maker. These
alternatives also demonstrated the additional insight to be gained from the VFT process.
When generating alternatives, two approaches were taken. First, a set of alternatives was
generated which represented the baseline or current system with improvements. These
alternatives show future iterations of the system with a product focus, in which new products are
added to the architecture with the intent of improving the VDEA score. The second set of
alternatives represents random scoring scenarios. These scenarios do not take into account the
current or baseline architecture.
Table 4.5 shows the alternatives that were generated for the decision-maker. These
alternatives represent different versions of the architecture. In Table 3.8, each measure is linked
to its source views. The “Perfect Architecture” represents an architecture that scores 100% value
in all currently available System Effectiveness areas as an upper bound. The “Current
Architecture” or “Baseline” is the architecture as provided by the sponsor. Other alternatives
represent possible alterations to the existing architecture, based on the addition of more views.
The views used for additional alternatives were determined based on the sources for measures
(Table 4.6) and the gap analysis shown in Figure 4.5. By determining where the current
architecture lost value, it was possible to generate alternatives that filled those gaps by adding the
necessary views. For example, the OPERATIONAL NEEDS measure requires at a minimum an AV-1
and SV-5, in addition to all existing views, to earn full value. Therefore, an alternative was
created for a full AV-1 and SV-5, which assumes that these products include the minimum
information necessary to assign full value to OPERATIONAL NEEDS.
Table 4.5. Generated Alternatives
Perfect Architecture
Baseline
Baseline plus OV-3
Baseline plus SV-5
Baseline plus SV-7
Baseline plus SV-8
Baseline plus SV-9
Baseline plus full AV-1
Baseline plus AV-1 and SV-7
Baseline plus AV-1, SV-7, and SV-5
Baseline plus SV-7, SV-8, and SV-9
Random VDEA Score 1
Random VDEA Score 2
Random VDEA Score 3
Random VDEA Score 4
Table 4.6. Required Views for Measures (Osgood, 2009)
alyoug
sprepuers
jeo/UY99!
T-AL
yseoa
404
ASojouysa
|
sadiAsas/suiaysAs
6-AS
uondiussag
UOIINJOAJ
Sad|AJaS/sWAaysAS
g-AS
XUJe|
SYayaWeJeg
BUeULOLIag
saoimias/swayshs
-AS
xe
Ayypiqeases
|
seaiAlas
/suiaysAs/uonsun4
wiayshs
0}
AWAY
|eUOIZe1adQ
S-AS
uondussag
uoeslunWWOD
Sad1IMaS/WAaySAS
Z-AS
uoldiuosag
ae]
-quangz/UOIsuel|
2}2]S/|@PON
sajny
jeuoessdo
9-AO
[apo
Ayanoy
jeuoiesado
S-AO
yey
diysuonejay
jeuoneziuesic
AO
xe)
BBueyoxy
uoneWojul
|euo!yesadQ
€-AO
uondiuosag
AyAnoauU0D
@pon
Jeuoesado
7-AO
aiydeig
ydaou0)
Jeuoiqe
edo
[9497-431
T-AO
Uo!eWJOJU]
Arewuuns
pue
Malang
T-AV
Operational
Needs
Threat
Detection
Threat
Assessment
Warning
Plan
Technological
Availability
Environmental
Impact
Monetary
Practicality
-
Maintenance
Monetary
Practicality
-
Initial
Supportability
Requirements
Adaptation
Reliability
Requirements
System
Redundancy
Recoverability
Requirements
Joint
Operations
NES!
Develop
ment
NESI
Evaluation
Suen
|
Ayiqeyoddns
Span
[PUONEISdO
»
PAPA
Ie
AEJQUOW|
»
0
yeasyl
Aqeljay
»
ISAN
§
WlOry,
JESU
Aduepunpay
»
quawidojanaq
ISAN
i
JEJUSWUUOIIAUZ
‘0
jea!3ojouysa!
»
sjuawalinbay
AyjiqeiaAooay
»
DUBUDIUARYPIVIEA
ALeJOUO||
5
oneqdepy
5
0
wn
3
Wn
UdAQIaYq
WaIsAsS
Figure 4.5. Gap Analysis Baseline plus SV-7 (in addition
to existing products) is an alternative created to show the
difference in score if the Maintenance value measures (except
SYSTEM REDUNDANCY) were assigned full values. Without an
SV-7, it is very difficult to score these measures. The OV-3
alternative exists since it is a supporting view for four
measures. Baseline plus SV-5 demonstrates the effect of
adding only an SV-5 without a full AV-1. Baseline plus SV-8
shows the value associated with adding an SV-8 and Baseline
plus SV-9 shows the value associated with adding an SV-9.
The SV-8 and SV-9 products account for the TECHNOLOGICAL
AVAILABILITY and ADAPTION measures. An alternative was also created that incorporates a full
AV-1, SV-7, and SV-5, which allows the hierarchy to maximize the OPERATIONAL NEEDS
measure, as well as three of the Maintainability measures. Finally, an alternative was generated
that includes an SV-7, SV-8, and SV-9. This alternative represents a situation in which
TECHNOLOGICAL AVAILABILITY, ADAPTATION, and three of the four Maintainability measures.
The full scoring scheme for each of these alternatives is shown in Appendix D.
The second set of generated alternatives was produced by allowing each measure’s score
to take on a random value. In cases where the measurement was on a continuous interval, a
random number was selected on that interval. In cases where the measure could take on discrete
categorical values, each category was given equal likelihood of occurrence for selection. Once
all scoring scenarios were generated, the related value was summed to obtain an cumulative
score. This process was performed 500 times and four of these 500 cases are presented as
alternatives. Since measurements are independent and mutually exclusive, each measurement
score may take on any scoring value without an effect on the measurement of other values. These
alternatives produced from Monte Carlo sampling are intended to provide additional comparison
criteria for the baseline.
Monte Carlo generation of alternatives was completed using a spreadsheet. To obtain the
composite random scores, the Architecture Quality scores were held static with their current
values. A Monte Carlo sampling technique was used to randomize System Effectiveness
measurements. This measurement was then transformed to a value using the SDVF and
multiplied by its measure weight ( ). The composite score was for each alternative was
calculated by summing all values for each measure. Five hundred separate scoring iterations
were created. These 500 new “alternatives” represented 500 different possible alternatives with
random measurements for each measure, and therefore a random score. They do not represent the
current JFPASS value or measurement, only hypothetical VDEA scores. Once the set of 500
scores were produced, a random number generator was used to select four of the iterations from
the set of 500. Each of the numbers selected represented a random score of the VDEA
instrument. The random selector chose random alternatives 26, 207, 379, and 420l; hereafter
called Random VDEA Score alternatives. Each of these situations were then entered to the
Hierarchy Builder© software and analyzed alongside the other generated alternatives.
4.2.5 Alternative Analysis
As a part of the JFPASS deterministic analysis, each of the generated alternatives
were also examined. Appendix D shows the scores for each of the measures in each
alternative. Figure 4.6 shows the rankings and scores of all alternatives. It is important to
note that though the addition of certain views may not help the overall score of the system,
that does not mean that those views at not useful. Each view within DoDAF serves a specific
purpose and should not be discounted if the information contained within it is helpful to the
system designers or decision-maker. This analysis is based on the information that is
important the scoring of the system based on the decision-maker’s values. If additional views
are required, they may not impact the score of the system, but this does not lessen their
importance to system design.
Figure 4.6 All Alternatives and Scores
The Perfect System score represents a system in which all System Effectiveness measures
have scored their maximum value. The Architecture Quality scores were held constant from
Cotton and Haase’s (2009) evaluation of the system. With all System Effectiveness scores
maximized, the system scored 0.890 of 1.000 possible value units.
The highest scoring “non-perfect” alternative was a situation in which SV-7, SV-8, and
SV-9 products are added. The addition of these views allows for the maximization of
TECHNOLOGICAL AVAILABILITY, ADAPTATION, and three of the four Maintainability measures.
Each of these views maximizes different areas of the hierarchy, to show the additive advantages
of creating additional views. This alternative scored 0.709 of 1.000 possible value units.
The next highest scoring alternative was the current system with and AV-1, SV-7, and
SV-5 added. Adding these views allows the system to maximize both the OPERATIONAL NEEDS
measure as well as the Maintainability measures. The addition of a full AV-1 and SV-5 allows
for OPERATIONAL NEEDS to be maximized, while the SV-7 allows for the Maintainability
measures to be maximized. This alternative scored 0.704 out of 1.000 value units. The
Random VDEA Score 4 alternative represents a random scoring situation. The individual scores
for each measure of Random alternative 4 are shown in Appendix D. Random alternatives were
generated to demonstrate to the decision-maker additional possible scoring situations which may
not have been considered in the generation of other alternatives. This allows the decision-maker
other means to achieve a certain level of architecture value. JFPASS with SV-7 was the
next highest scoring alternative. This alternative is meant to demonstrate the effect that the
addition of an SV-7 that includes the necessary documentation for the Maintainability measures
would have on the system score. SV-7 is therefore the next most important view to be added to
the existing architecture. This view would provide the required information to prove that the
values under Maintainability were considered in the system design.
The score for this measure was 0.663.
Random VDEA Score 3 scored the next highest value. This alternative scored 0.583. The
JFPASS alternative that includes AV-1 and SV-5 allows the OPERATIONAL NEEDS measure to be
maximized. By adding these views with the required information, the score of the system
becomes 0.578. The SV-8 product allows the system to maximize its score for ADAPTION. If this
view is created, with necessary information for an improved ADAPTION score, the total system
score increases from 0.538 to 0.565.
SV-9 allows for a similar increase in value. The addition of an AV-9 view will allow for
the TECHNOLOGICAL AVAILABILITY measure to be scored higher. This measure is more difficult
though. There is a possibility that simply adding the SV-9 product will not increase the score.
Since the TRL is based on the actual availability of technology, in order to maximize the TRL
scoring, the technology used in the system must rate higher on the TRL scale. The creation of this
alternative allows the decision-maker to see how much of an effect the maximization of
TECHNOLOGICAL AVAILABILITY will have on the total system score.
Random VDEA Score 2 scored the next highest on the list of alternatives. This random
alternative had a score of 0.540 and represents a system with very minor changes to the current
configuration. In addition, this alternative shows the effects of failing to maximize or earn value
on some of the views that the baseline has achieved value on.
The next three alternatives all scored the same as the current JFPASS system. The
addition of an SV-5 product by itself does not add any value to the system. The SV-5 is used to
trace capabilities to system functions, but without a list of OPERATIONAL NEEDS, the SV-5 does
not add much value based on what is important to the decision-maker. The SV-5 is only useful
when combined with a complete AV-1. Without the SV-5, though, a more complete and explicit
AV- 1 is of little value as well. The system with an OV-3 added also does not earn any
additional value. This was the last view that could possibly add value to the system or was
useful in the determination of the scoring of some values. The OV-3 is used for the score of
OPERATIONAL NEEDS, THREAT DETECTION, THREAT ASSESSMENT, WARNING PLAN, and JOINT
OPERATIONS. Since the OV-3 is only a supplemental view for determining the score of these
measures, its additional does not help the current system. In the case of THREAT DETECTION,
THREAT ASSESSMENT, WARNING PLAN, and JOINT OPERATIONS, these measures were already
scored at maximum value, therefore the OV-3 does not assist in scoring those measures.
Finally, the Random VDEA Score 1 Alternative has a VDEA-Score of 0.454. This random
alternative represents a system which is more lacking than the existing system. This random
alternative lacks many of the stronger valued measures and therefore scores lower. This
alternative shows a contrast to the baseline in which many of the values did not earn their full
values.
The scores and difference from the current system for all alternatives can be found in
Table 4.7. This table illustrates exactly how much effect improvements to the current architecture
will have on the total score. For example, the addition of an SV-7 can cause a maximum change
of 0.125 value units to the system. Therefore, adding this one product has the highest impact on
the system. Adding an SV-7, SV-5, and completing the AV-1 will have a maximum change of
0.166 value units. But making this change will require the addition of two products and
alterations to an existing product. Therefore, the most cost effective value adding measure will be
to add an SV-7.
Table 4.7 Alternative Scores and Maximum Value Additions
Alternative Name
Score
Maximum
Value Change
JFPASS Perfect System
.890
0.352
Baseline plus SV-7, SV-8, and SV-9
.709
0.171
Baseline plus AV-1, SV-7, and SV-5
.704
0.166
Random VDEA Score 4
.686
0.148
Baseline plus SV-7
.663
0.125
Random VDEA Score 3
.583
0.045
Baseline plus AV-1 and SV-5
.578
0.040
Baseline plus SV-8
.565
0.027
Baseline plus SV-9
.557
0.019
Random VDEA Score 2
.540
0.002
Baseline
.538
0.000
Baseline plus full AV-1
.538
0.000
Baseline plus OV-3
.538
0.000
Baseline plus SV-5
.538
0.000
Random VDEA Score 1
.454
-0.084
4.3 Sensitivity Analysis
The Hierarchy Builder© software includes a sensitivity analysis tool which examines the
weight of each value and measure in the hierarchy. This sensitivity analysis tool allows the user
to see how the VDEA score would be changed by adjusting the weight of a particular value or
measure. By examining how weights affect the scoring of alternatives, the decision-maker can
gain insight as to how the weighting of a certain value affects the decision.
4.3.1 Global Weight Sensitivity Analysis
The Hierarchy Builder© software’s sensitivity analysis tool is capable of performing
sensitivity analyses both globally and locally. This sensitivity analysis shows the decision maker
how the final decision may be affected by altering the weight of a particular value. Since the
weights were chosen and validated by the decision-maker, the current weights are assumed
correct. Sensitivity analysis is provided to demonstrate alternative scenarios and allow for further
verification of weight.
4.3.1.1 Alternative Sensitivity Analysis
Sensitivity analysis in a multiple alternative situation is generally more useful to the
decision-maker. In a multiple alternative situation, adjusting the weights of certain values or
measures may lead to one alternative being chosen over another. In some cases, a small change in
weight may lead to a major change in the ranking of alternatives. It is therefore important to
closely examine the sensitivity curves for each of the alternatives to ensure that the weighting is
correct and the proper alternative is being selected. In the case of the JFPASS, the alteration of
some weights may affect which product should be included next in the architecture. With more
alternatives, the complexity of the analysis and decision increases, and minor alterations may
have a larger effect on the outcome. Any changes to weights must be well vetted through the
decision-maker to ensure the change is being made for the proper reasons.
4.3.1.1.1 Alternatives System Effectiveness Sensitivity Analysis
When a sensitivity analysis is performed on all alternatives, the best performing system
is, quite obviously, the Perfect system alternative. This provides a good point of comparison for
the other alternatives. The analysis shows that if the local weight of System Effectiveness is
increased to one, the perfect system is by far the best performing alternative; this is followed by
the SV-7, SV-8, and SV-9 alternative. The next best performing alternative is the Current system
plus AV-1, SV-7, and SV-5 included. Though this is the next best performing alternative, there is
still a great deal of value not being earned. Each of the alternatives has a negative slope as the
weight of System Effectiveness approaches one. This chart shows the decision maker that even in
the case of the best generated alternative, there are value gaps. In addition, if the global weight
of System Effectiveness is altered between zero and one, there will be no effect to the alternative
preference. Figure 4.7 shows the sensitivity analysis for System Effectiveness with all
alternatives included.
Figure 4.7 System Effectiveness Alternative Sensitivity Analysis
4.3.1.1.2 Alternatives Capability Sensitivity Analysis
When the Capability value is examined, there are significant changes to the alternative
ranking that can be affected by changing the weight of the Capability value. The Perfect system
will of course continue to perform the best of all alternatives as the weight increases. As the
global weight of Capability approaches approximately 0.4, the Random VDEA Score 4 alternative
begins to outperform the AV-1, SV-7, and SV-5 scenario. As the global weight of
Capability increases or decreases, it may affect the decision that will be made. Depending on the
alternative chosen, the amount of adjustment before the decision is impacted alters. In the case of
Capability, if the weight is incorrect, it will not change the decision from this point. These rankings,
as well as the points at which the alternatives change order can be seen in Figure 4.8. If the weight
is set to any value between zero and one, a new ranking can be found by observing the order of
alternatives. Locally, there are similar changes to the rankings of alternatives when examined.
Figure 4.8 Capability Alternative Global Sensitivity Analysis
4.3.1.1.3 Alternatives Maintainability Sensitivity Analysis
As the global weight of Maintainability increases, the ranking of alternatives changes
significantly. Several alternatives’ values drop to below 0.1 in fact. The range of alternative
scores varies between 0.05 and 0.82 (without consideration of the Perfect System).
Capability however only varies between 0.4 and 0.78. This shows that the current scoring of
Maintainability offers a great deal of opportunities for improvement. It is also evident that in
some alternatives, particularly in the mid 0.6 value range, the decision is very sensitive to the
weighting of Maintainability. The alternatives in which the Maintainability measures have
attained their maximum value perform much better in a Maintainability sensitivity analysis.
The random alternatives are the only alternatives that score differently than the generated
alternatives. All other generated alternative either rank in the lower or higher group, due to
the effect of the Maintainability measures. The generated alternatives generally perform
together, since the addition of the SV-7 product allows all Maintainability measures to
maximize. The random alternatives are not bound by all Maintainability measures
performing together; therefore, they score differently. Locally, the alternatives perform
similarly to globally, but the range of values decreases. Figure 4.9 shows the global
sensitivity analysis for Maintainability.
Figure 4.9 Maintainability Alternative Global Sensitivity Analysis
4.3.1.1.4 Alternatives Interoperability Sensitivity Analysis
The sensitivity analysis for Interoperability shows that again the random alternatives are
the only one that performs significantly differently. All other alternatives converge to the same
value of approximately 0.6 as their global weights approach one. As the weights approach zero,
their values diverge slightly from their current state, but the rankings do not change. This implies
that in order to affect any significant change to the final score, the Interoperability measures must
score differently than they do in the majority of the alternatives. In this case, the weight of
Interoperability can have a major effect on the decision. If the weight of Interoperability is
increased to 0.5, the SV-5 is the next best choice. Until that point, the decision remains
unaltered. In an analysis of the scoring of each alternative, it is apparent that each of the random
alternatives has differing scores for the Interoperability measures, causing their changes in the
sensitivity analysis. Figure 4.10 shows the global sensitivity analysis for the Interoperability
value.
Figure 4.10 Interoperability Alternative Global Sensitivity Analysis
Chapter 5. Conclusions and Recommendations
This chapter provides an overview of the research completed and the results of the
JFPASS system analysis. The research questions for this effort were answered by leveraging the
Value-Focused Thinking (VFT) process. Recommendations for the existing architecture as well
as future architecture developments in the Joint Force Protection Advance Security System
(JFPASS) project were also determined. In addition, suggestions for improvement of the
ValueDriven Enterprise Architecture (VDEA) tool are discussed.
5.1 Evaluation Result
The research questions for this effort were “(1) Can the VFT process be applied to an
evaluation of a set of architectural products? (2) What is the resulting Hierarchy to evaluate a
force protection system? (3) What are the related weights and measures for the hierarchy? (4)
What score does the provided architecture score based on this hierarchy and where are the
shortfalls and potential areas of improvement?” Each of these questions were answered during
this effort.
It was shown that the VFT process could be applied to evaluate a set of architectural
products. Through research into existing guidance and documentation and interviews with
decision-makers, it is possible to determine what values are important to those decision-makers in
architectural design. Through the use of the VFT process, a Value-Driven Enterprise Architecture
methodology was found. Even with a single alternative, the VFT process was able to output a
score for the system as a whole to be used as a baseline for future improvements. In addition to
giving the decision-maker a baseline, the VFT process allowed for the creation of alternatives
that could show the possible future maturation and development of the project.
These alternatives gave the decision-maker a set of comparison criteria to determine the
future direction of project development.
The resulting hierarchy for JFPASS evaluation was developed in two major branches: the
System Effectiveness Branch and the Architecture Quality branch. This effort determined a
possible hierarchy for System Effectiveness evaluation. This Hierarchy is shown in Figure 5.1.
Figure 5.1 System Effectiveness Hierarchy
In addition to the hierarchy, weights were assigned to each value and measures of
effectiveness assigned to each lowest tier value. The weights assigned to each value were
determined through both research of the guidance and documentation and interviews with subject
matter experts (SMEs) in the field of force protection (FP). The resulting 16 measures allow the
decision-maker to determine the effectiveness of the system in question.
Finally, the JFPASS was assigned a score of 0.538 of 1.000 possible value units through a
deterministic analysis. This score includes the value earned by both the System Effectiveness and
the Architecture Quality. This is also an “earned value” as opposed to a final score. At this point
in time, the JFPASS has earned 0.538 of its possible value units. As it is still early in its
development, there is time to earn additional value for this system and improve the score. This
score also serves to show the level of development of the current system, as well as which areas
are lacking. The shortfalls and suggested areas of improvement are presented in the next section.
5.2 Recommendations
Through deterministic analysis and sensitivity analysis of the system, several
recommendations were generated, both for improvement of the current system and for future
development of the system. These recommendations are intended to help guide work on the
JFPASS project with the final system value in mind. Final determination of the course of future
system development lies in the hands of the decision-maker and system sponsor, but the scores
and sensitivity analysis provide a justification for possible changes to system development.
5.2.1 Future View Development
Several areas of evaluation in System Effectiveness require more information or additional
views to properly score. The addition of these views would allow for the scoring of certain
measures of effectiveness, providing more value to the overall system. Through a sensitivity
analysis of the current measures, it was possible to determine the maximum benefit provided by
each measure and view. Table 5.1 shows the maximum benefit that the creation of each new
view would have on the total score.
The JFPASS Perfect System alternative exists for decision-maker comparison, not as a
practical alternative. The highest scoring realistic alternative is one which incorporates a
complete SV-7, SV-8, and SV-9. The next alternative in the ranking requires three additional
views, each of which builds on the information in others. By creating these additional views in a
specific order, value can be added more quickly. For these recommendations, the random
alternatives are not considered. They are included in the analysis as a basis of comparison for the
decision-maker. These recommendations may be applied to cost-benefit analysis.
Table 5.1 Maximum Value Benefit of New Views
Alternative Name
Score
Maximum Value
Benefit
JFPASS Perfect System
0.890
0.352
Current System plus AV-1, SV-7, and SV-5
0.704
0.166
JFPASS Random 4
0.686
0.0148
Current System plus SV-7
0.663
0.0125
JFPASS Random 3
0.583
0.045
Current System plus AV-1 and SV-5
0.578
0.040
Current System plus SV-8
0.565
0.027
Current System plus SV-9
0.557
0.019
JFPASS Random 2
0.540
0.002
Baseline
0.538
0.000
Current System plus Full AV-1
0.538
0.000
Current System plus OV-3
0.538
0.000
Current System plus SV-5
0.538
0.000
JFPASS Random 1
0.454
-0.084
Based on this analysis, the next most beneficial product to create would be the SV-7,
System Performance Parameters Matrix. This view allows the architect to add information
regarding the parameters to which system components are designed. This includes three of the
four Maintainability measures required for this evaluation. The information contained in the SV-
7 product may already be used in the system design, but without the SV-7, there is no other place
in the architecture that the information can be found. For the system to be properly designed, it
must have some parameters by which system components are designed, specifically regarding
their supportability, reliability, and recoverability. These ideas must simply be included in the
architecture to ensure compliance with the parameters and proper representation to the reader. By
simply adding the SV-7 product, the total system score has the potential to increase by 0.125
value units (if all required information is included).
Following the creation of the SV-7, an SV-5, Operational Activity to Systems Function
view would create the next highest value benefit. The SV-5 is intended to show the reader which
components are accomplishing which functions. By tracing those functions up to operational
requirements, it is possible to determine how the operational needs of the system are being met,
i.e. using which system components. For the SV-5 to be of any benefit, though, the AV-1 must
also be updated with an explicit list of the operational requirements for the system. The AV-1
currently includes a discussion of the purpose of the system, but lacks any specific discussion of
the problems that the system will solve and the constraints by which the system is being designed.
It was of great importance to the decision-maker that the system accomplishes its goals.
Therefore, those goals must be outlined explicitly. The AV-1 provides the best context for this
discussion. These details may also be included in some appendix to the architecture, but should
be included with the architectural package. Without a list of operational needs or requirements,
the SV-5 is of no benefit to the system. If the SV-5/AV-1 update is completed following the
creation of the SV-7, the three updates have a positive cumulative effect on the system score.
They will cause a positive change of 0.166 value units to the final score.
The next most beneficial view would be the SV-8, Systems Evolution Description. The
inclusion of the SV-8 allows the architecture evaluator the ability to score the ADAPTION measure.
There are currently no details included in the architecture regarding the Flexibility of the system.
The SV-8 has the ability to show the reader how the system may evolve not only further into the
acquisition and design process, but under operational constraints. The inclusion of an SV-8 with
the necessary information to score Adaptability may cause a benefit of 0.027 value units.
Finally, the SV-9, Systems Technology Forecast view would provide the next most benefit
to the system score. The inclusion of the SV-9 has the ability to add 1.9 value units to the total
score. The SV-9 product allows the reader to determine the current Technology Readiness Level
(TRL) of the components included in the architecture. The TRL of each component is required to
determine the overall system TRL for the TECHNOLOGICAL AVAILABILITY measure. The
OV-3, while included as supporting documentation for some measures, is not required for this
system. It is possible to determine the information included in the OV-3 without its inclusion for
scoring. The OV-3 may include other information of use to the architect or decision-maker,
therefore its inclusion should not be complete discounted. The architect is also bound by the
milestone decision point requirements for architecture products. In other force protection
systems, this view may be required to score the architecture. The addition of an SV-5
alone will not add any value either, unless it is added with the AV-1 updates. Conversely, the
addition of a complete AV-1 will not have any positive effect on the system score without the SV-
5.
5.2.2 System Strengthening
In addition to future development of views, several steps may be taken to strengthen the
current system and its score. In some cases, more value may be earned by improving upon
information already included or updating design decisions based on the decision-maker’s most
important values. In other cases, a measure may already score full value, but the inclusion of
additional information may make the architecture more easily scored and read.
The THREAT DETECTION, THREAT ASSESSMENT and WARNING PLAN measures were all
scored positively. Though they were scored at their highest level, it is possible to make them
more easily accessible to architecture readers. Each of these measures refers to the inclusion of a
plan related to the measure. To determine the degree of attainment of these measures, the OV-5
product was used. Since the activities required to accomplish each of these concepts were
included, they were scored positively. Including actual text versions of the plans may assist not
only the architecture readers, but also future users of the architecture. Each installation is
required, regardless of service component to have official, written plans for these concepts. The
inclusion of skeleton versions of these plans in the architecture would assist in the scoring and
ensure compliance of each installation.
The ENVIRONMENTAL IMPACT measure was scored “CONUS and Contingency
constratints.” This measure is capable of earning 0.2 additional value units by adding Host
Nation constraints to the current consideration. Including international environmental policy
documents in the TV-1 would allow this measure to be scored at a higher level. It may also be
possible to include the environmental policies of several nations that the U.S. military historically
operates and has standing military commitments (or Status of Forces Agreements (SOFA)) with.
The MONETARY PRACTICALITY measures refer to the budgetary constraints that all
government programs are subject to. The ability to construct a system within budget is of major
concern to all stakeholders in a project. The inclusion of specific cost information to the
architecture would not only allow the scoring of these measures, but would add verification to the
stakeholders of a system’s fiscal viability. A total system estimate and program budget must of
course be included as well in order to compare the cost information to. The OV-5 product has the
ability to display this information. Initial cost and life cycle costs may also be included in the
AV-1 product, although the costs would not be itemized by component.
The two Net-Centric Enterprise Solutions for Interoperability (NESI) measures also have
room for improvement. Currently, the NESI DEVELOPMENT measure is scored positively, but its
assessment may be improved by explicitly including the NESI documents in the TV-1. The
assessment for this measure was done by comparing the included system design and design
documents to the NESI guidance and determining that the system was being constructed with Net-
Centricity in mind. Including the NESI documents in the architecture would show the
architecture reader that the system was in fact designed with these concepts in mind. In addition
to including the Net-Centricity documents, a NESI evaluation should also be completed on the
architecture. Simply completing the evaluation would allow for a positive score of the NESI
EVALUATION measure, but inclusion of the evaluation in the architecture, as well as a positive
score would assist in the scoring of the architecture.
5.3 Model Strengths
The creation of the Value Driven Enterprise Architecture tool allows a force protection
architecture to be objectively evaluated. This tool gives the decision-maker an objective
numerical score from which to base future revisions and additions to the architecture. This
baseline combined with analysis of the score shows the most beneficial views to be created in the
future and improvements that may be made to the existing architecture.
In interviews with several force protection experts, the system was found to be all
inclusive of the important values for force protection. Each of the SMEs found no major areas of
force protection that were not included in the values of this model. By creating a system built
around the values alone, a comprehensive force protection system may be constructed. This
model allows a project stakeholder or sponsor to “score” a system based solely on its
architecture. This is useful since many acquisition decisions are made based solely on
architectural products. Having a tool to evaluate them allow for objective evaluation of the
architecture.
