new acquisition processes in DoD Instruction 5000.02

Proceedings of the 2013 Industrial and Systems Engineering Research Conference

A. Krishnamurthy and W.K.V. Chan, eds.

Effect of the Analysis of Alternatives on the DoD Acquisition System

Eugene Rex L. Jalao; Danielle Worger; Teresa Wu, PhD

Arizona State University

Tempe, AZ, 85281

J. Robert Wirthlin, PhD; John M. Colombi, PhD

The Air Force Institute of Technology

Wright-Patterson AFB, OH 45430

Abstract

The Enterprise Requirements and Acquisition Model (ERAM) is a discrete event simulation that models the major

tasks and decisions within the DoD acquisition system. A majority of DoD acquisition projects are being completed

behind schedule and over budget. ERAM suggests process improvements can have salutary effects. Hence,

enhancements in improving the end-to-end acquisition process would be worthwhile. Until 2008, the Analysis of

Alternatives (AoA) process is a mandatory task for acquisition category (ACAT) level 1 projects. As such expected

program completion time for ACAT 2 and ACAT3 categories is shorter. Since 2008, the AoA became a required

procedure for all programs. However, to the best of our knowledge, the impact of requiring all programs to complete

an AoA has not yet been studied in literature. This research addresses this gap with two main contributions. First,

this research seeks to quantify the amount of delay on total completion time when the AoA is required for all ACAT

programs. Secondly, the sensitivity of the processing time and variability of the AoA process is simulated and its

effect is studied on total program completion time. Viable policies and intervention strategies are then inferred from

these contributions to further improve acquisition program completion time.

Keywords DoD, Acquisition, Simulation, Analysis of Alternatives

1. Introduction It is a known fact that a large number of Department of Defense (DoD) projects are being completed behind

schedule and over-budget [1]. A Government Accountability Office (GAO) report released in 2009 states that for the

DoD’s 2008 portfolio, on average a program faced a 22-month delay and exceeded the original budget [2].

Generally, total cost growth has been consistent over the past few decades with a recent assessment by [3] of 1.44 or

44% growth. The current DoD Acquisition system which is composed of three separate and distinct processes,

including the Joint Capabilities Integration Development System (JCIDS), the Planning, Programming, Budgeting &

Execution (PPBE) process, and the formal acquisition development system outlined by the DoD 5000 series of

instructions, does not exist in a static environment. The system is constantly being adjusted, either through policy

changes or statute [4-6]. Since the acquisition process is a large, complex, socio-technological system, it is difficult

to determine which processes or factors affect performance metrics like time, cost, and resource utilization. Hence,

alternative modeling tools to improve the DoD acquisition process are the subject of current research.

In 2009, a discrete event simulation (DES) model called the Enterprise Requirements and Acquisition Model

(ERAM) is developed by Wirthlin [7]. This model simulates the actual acquisition processes of the US DoD using

the Air Force implementation of acquisition processes for Acquisition Category (ACAT) levels as the basis of the

model. This is done in order to provide further insight and understanding of the complex system’s behavior.

However, this research did not include new policies set forth by the DoD, specifically the DoD Instruction 5000.02

that requires all programs to go through an Analysis of Alternatives (AoA) process [6] before reaching Milestone A.

The AoA is a requirement for all military acquisition programs. By definition, the AoA is an analytical comparison

of multiple alternatives process that needs to be performed prior to committing resources to a given acquisition

program [8]. According to the DoD 5000.02 instruction, “The AoA shall focus on identification and analysis of

2110

Jalao, Worger, Wirthlin, Colombi, Wu

alternatives, measures of effectiveness, cost, schedule, concepts of operations, and overall risk. The AoA shall assess

the critical technology elements (CTEs) associated with each proposed materiel solution, including technology

maturity, integration risk, manufacturing feasibility, and, where necessary, technology maturation and

demonstration needs.” Through this requirement, an implicit assumption is being made that this step will actually

shorten the overall lifecycle development time for a given acquisition program and increase the quality of the final

form of the materiel solution. However to the best of our knowledge this policy effect has not yet been

quantitatively studied in existing literature. Nevertheless, the AoA theoretically contributes to longer program time

completion of all DoD acquisition projects as it is an additional task that must be performed during the process.

Hence potential policies could be developed to counter the effect of requiring the AoAs, e.g. acknowledging better

quality solutions earlier in development, which could be easily translated to viable policies to further improve not

only the duration but the entire end-to-end DoD acquisition system.

Against this background, this research addresses these limitations by performing additional simulation and statistical

analysis on the ERAM model. The primary goal of this research is to determine the effect on the total acquisition

program completion time by requiring all ACAT programs to go through the AoA process. Furthermore, the effect

of reducing the variability and the time spent on the AoA process is studied and potential intervention strategies are

developed. Additionally, this research provides viable policies and discussion points from the intervention strategies

that could further reduce program completion time. The rest of this paper is organized as follows. Section 2

provides an overview of the related literature on the AoA, while Section 3 provides the methodology for the

analysis. Section 4 presents the results of the study while section 5 provides conclusions and future research options.

2. Review of Literature

2.1 DoD Enterprise Requirements Acquisition Model (ERAM)

ERAM is originally an Arena simulation model which provides the foundations for research on applying discrete

event system simulation to the DoD acquisition process. Extensive validation of the model is done by comparing the

performance results of the model to the actual DoD acquisition data and expert reviews. Initially, 20 intervention

strategies are explored using the ERAM model to assess the potential of the total completion time reduction of DoD

acquisition projects. If all 20 interventions are implemented, a 20% reduction in the total program time could be

realized. It is found that the most effective interventions to improve the system are those that reduce the variability

of the processes. Since its publication in 2009, the Arena model has been translated to ExtendSim and extended by

The Aerospace Corporation’s Developmental Planning and Architectures Division for use in the Concept

Development Center of the Space and Missile Systems Center at Los Angeles AFB, CA [9]. Moreover, Montgomery

[10] provided the research for Aerospace to extend the model to further include ACAT 2/3 programs along with

modeling the Rapid Acquisition process for space programs. Table 1 summarizes the different versions of the

ERAM model. Please note the earlier version of ERAM (1.0) does NOT implement AoA on all ACAT projects.

Table 1: ERAM Versions Adapted from [11]

Author Version Number Changes

Wirthlin [7] ERAM 1.0 Baseline Translation from Arena to ExtendSim

Leach and Searle [9]

ERAM 1.1 Updates by the Aerospace Design Team and Served as new

baseline model

ERAM 1.2 Implemented new DoD 5000.02 policies

ERAM 2.0 Incorporated the global variables that modify acquisition

capabilities

ERAM 2.1 Incorporated the JCIDS review process

Montgomery [10] ERAM 2.2 Added more capabilities for ACAT 2/3 and Rapid Acquisition

Process

Verification and validation of the baseline distributions included hand modeling, iterations of correction from

feedback of experts in all three branches of acquisition, and comparison of schedule and budget information from

the DAMIR and SMART databases to distributions of the schedule time of model-generated data [7].

2.2 Implementation of the Analysis of Alternatives

There are also several articles that illustrate the implementation of the AoA in the DoD acquisition process.

Cervantes et al. [12] applies a rapid AoA implementation to find a NATO Special Operations Headquarters (NSHQ)

2111

Jalao, Worger, Wirthlin, Colombi, Wu

Air Wing. Georgiadis et al. [8] propose an analytical multiple criteria decision making methodology to handle the

AoA. The AoA also plays a part in the replacement of the aging presidential helicopter fleet [13].

Research also explores policies to improve the implementation of the AoA in the end-to-end DoD acquisition

system. For example, Schank [14] identifies several important factors for the success of the AoA specifically: (1) the

AoA must have a study plan that considers a wide range of alternatives and must be flexible in the analysis

methodology (2) oversight committees that manage effect relationships, (3) trade-off analysis should be conducted

on all alternatives, and (4) good estimation and recognition of technical, design and production risks is a must. A

white paper from the Training and Doctrine Comman (TRADOC) Analysis Center (TRAC) addresses two problems

of the implementation of the AoA specifically: (1) by clearly describing and differentiating the purpose and scope of

each AoA for each milestone decision, (2) by clearly defining and describing “materiel solution” as it can be used

interchangeably with the “alternatives” term [15]. Stadterman et al. [16] propose improvements on the AoA in the

Weapon Systems Acquisition Reform Act of 2009. Among these recommendations are: (1) all concerned parties

within the AoA should build a working relationship, (2) the AoAs should focus on the decision choices and the

decision space, and have an achievable, affordable and operationally relevant set of criteria and (3) the Army should

follow a formal analytical process that support the AoA throughout the acquisition process. Additionally, Ford et al.

[17] improve the implementation of the AoA by incorporating benefits in the decision methodology through the use

of a system dynamics model of a military operation and integrating it with a Knowledge Value Added methodology.

Furthermore, in a Government Accountability Office (GAO) report [18], the GAO recommends that the Joint

Requirements Oversight Council should establish a review mechanism for AoA earlier in the acquisition process.

According to Roper [19], all alternatives being considered within the AoA must go through consistent analysis

methodologies and assumptions in order to ensure comparability. Hence, adopting these quality improvement

suggestions also imply a reduction in the length and/or the variability in the AoA process. Doing so will allow

AoAs to obtain consistent AoA process times.

Based on this review, the following gaps in literature can be gleaned:

• Currently, a quantitative study to assess the impact of requiring AoA implementation in all ACAT programs is lacking.

• Secondly, the effect of reducing the variability and length of the AoA process has not yet been addressed as a potential intervention to further improve the program acquisition completion time.

3. Simulation Analysis

This research is composed of three distinct phases. The first phase is a simulation study to determine the effect of the

AoA for ACAT 2 and ACAT 3 programs on the total completion time. The second phase consists of two simulation

experiments in which the variability and the mean of the AoA process time are reduced and its effect on total

completion time is determined. Lastly, the third phase is the translation of results into recommendations and further

research.

3.1 Requiring an AoA on a Percentage of ACAT 2 and 3 Programs

The original ERAM 1.0 model is utilized since it is the baseline model that did not require an AoA on all programs.

By varying the “ACAT 2 or ACAT 3 funding” process, specifically the percentage of programs with funding

already available for an AoA, the effect of requiring a certain percentage of programs to undergo an AoA on the

total program completion time, or the time the program completes Milestone C (MS-C), is determined. Figure 1

presents the screenshot of the module. This module decides if the program has enough funding to perform an AoA.

If it is found there is enough funding, the program will have to undergo an AoA.

ACAT 1 programs are not included in the analysis since AoAs are required previously. The baseline scenario is set

as such the probability that the program would be required to undergo an AoA is 1%. This setting is set before the

DoD policy change in 2008 on AoA implementation. Please note in [7], only ACAT 2 and 3 programs with

sufficient funding are needed to undergo an AoA. Simulation trials each with 3000 iterations are run separately for

ACAT 2 and ACAT 3 programs. During each trial, the probability that a program would need to perform an AoA is

increased incrementally, and its corresponding completion time is tabulated. Each increase is compared to the

baseline trial. A t-test is then performed in order to determine if there is a statistically significant difference between

the trial results and the baseline trial in terms of the time until MS-C. Hence, the probabilities are incrementally

increased until a statistical significant difference from the baseline is obtained at a 95% confidence level.

2112

Jalao, Worger, Wirthlin, Colombi, Wu

Figure 1: Arena Screenshot of “ACAT 2 or ACAT 3 funding” Process

3.2 Sensitivity Analysis on the AoA Process Parameters

The objective of this phase is to identify the impact of improving the AoA process specifically its variability and its

program completion time. The ExtendSim model is utilized over the Arena model because the model reflects the

most recent policy changes. The current AoA process is distributed according to a triangular distribution with a

lower limit of 180, most likely value of 360, and an upper limit of 720 days (Tria(180,360,720)), and all programs

are assumed to undergo an AoA. These settings constitute the baseline trial for this phase. The ACAT 1 level

projects are utilized in this phase. The ExtendSim simulation module being modified in the ERAM model is an

Activity block from the items library called “Analysis of Alternatives.” Figure 2 presents a screenshot of the

ExtendSim activity.

This phase is composed of two simulation experiments. The first experiment studies the effect of reducing the

variability of the AoA process time while the second section deals with reducing the mean of the AoA process time.

For the first experiment, 3000 iterations are performed and during each trial, and the variability of the time to

perform the AoA is adjusted. The variability is adjusted by reducing the difference between the mode and the

maximum/minimum by a constant. Hence, the triangular distribution would then have less variance. The variance of

the triangular distribution is reduced incrementally until a statistical significant difference is obtained from the

baseline trial. A t-test is then performed in order to compare the time to milestone C in the trial to the baseline trial.

Furthermore, the second simulation experiment deals with reducing the AoA process time to determine its effect on

the time ACAT 1 programs reach Milestone C. Three thousand iterations are again performed and for each trial, the

mean of the AoA program length distribution which is distributed according to a triangular distribution

(Tria(180,360,720)) is reduced. The ratio between the minimum, most likely, and maximum of 1:2:4 is maintained

as the parameters are changed. The mean completion time of the triangular distribution of the AoA is again reduced

until a statistically significant difference from the baseline trial is obtained. A t-test is again performed in order to

compare the time in the trials to the baseline trial.

3.3 Results and Interpretation

The final phase in this paper compiles the results from the tests and translates them into recommendations for viable

policy changes. This phase includes identifying which parameter adjustments improved the performance metrics and

making recommendations on further research, like interactions among parameters.

2113

Jalao, Worger, Wirthlin, Colombi, Wu

Figure 2: ExtendSim Screenshot of “Analysis of Alternatives” Activity

4. Results and Discussions Table 2 summarizes the results of the t-tests performed from section 3.1 for ACAT 2 programs. The table shows a

subset of trials corresponding to 50%, 85%, 87.5%, 90%, 95% or 99% of programs are required to undergo an AoA.

These settings are selected to show the sensitivity of changing the percentage of programs required to undergo an

AoA. These are compared to the baseline AoA setting in which only the funded programs (1%) are required to

undergo an AoA. The null hypothesis for the t-tests is ��:����� � ��% which corresponds to an insignificant

difference between the baseline and the ��� percentage if not rejected and alternative hypothesis ��:����� � � ��% if

there is significant difference.

Table 2: Summary of t-test Results of ACAT 2 Programs

% of Programs required to undergo an AoA

1%

(Baseline)

50% 85% 87.5% 90% 95% 99%

Average Time to

MS-C (Days)

3898.099 4001.104 4080.293 4093.409 4108.58 4115.981 4141.088

Standard Deviation

(Days)

1411.37 1550.684 1625.738 1634.814 1650.243 1655.492 1673.537

T-Value -1.34133 -1.78231 -1.90576 -2.04267 -2.04267 -2.34049

Conclusion Fail to

Reject H0

Fail to

Reject H0

Fail to

Reject H0

Reject

H0

Reject

H0

Reject

H0

It is evident that when requiring an AoA on at most 87.5% of the ACAT 2 programs will result in a failure to reject

that the ��� percentage is similar to the baseline scenario. Any percentage less than 87.5% will have no effect on the

average time a program arrives at MS-C. This means that a majority of the ACAT 2 programs can be subject to the

AoA and no significant increases in the total completion time can be obtained.

Table 3 summarizes the results of the t-tests performed for ACAT 3 programs. The table shows a subset of trials

corresponding to 50%, 55%, 57.5%, 60%, 65% or 75% of programs are required to undergo an AoA. Furthermore,

these settings are selected to show the sensitivity of changing the percentage of programs required to undergo an

AoA. These are compared to the baseline in which only the funded programs (1%) are required to undergo an AoA.

The null hypothesis for the t-tests is ��:����� � ��% which corresponds to no significant difference between the

baseline and the ��� percentage if not rejected and alternative hypothesis ��:����� � � ��% if there is significant

difference.

2114

Jalao, Worger, Wirthlin, Colombi, Wu

Table 3: Summary of t-test Results of ACAT 3 Programs

% of Programs required to undergo an AoA

1%

(Baseline)

50% 55% 57.5% 60% 65% 75%

Average Time to

MS-C (Days)

3334.926 3470.918 3493.278 3512.661

3524.566 3551.2 3604.434

Standard Deviation

(Days)

1143.19 1314.598 1350.852 1365.653 1372.66 1392.929 1419.016

T-Value -1.63342 -1.86606 -2.08392 -2.2198 -2.51299 -3.1095

Conclusion Fail to

Reject H0

Fail to

Reject H0

Reject

H0

Reject

H0

Reject

H0

Reject

H0

It is evident that when requiring an AoA on at most 55% of the ACAT 3 programs will result in a failure to reject

that the ��� percentage is similar to the baseline scenario. On the other hand, the AoA contributes to a significant

change in the total program completion time when at least 57.5% of the programs require AoA. In general, it is

evident from tables 2 and 3 that requiring an AoA on all programs significantly increase the total completion time of

ACAT 2 and 3 programs when programs are required to undergo an AoA.

Table 4 summarizes the results of the t-tests performed in terms of reducing the variability of the AoA process time.

Four settings are tested specifically, Tria(270, 360, 540), Tria(315, 360, 450), Tria(337.5, 360, 405), and Tria(360,

360, 360). These settings are selected to illustrate the sensitivity of the variability of the AoA processing time from

the baseline scenario to a deterministic scenario (time is fixed at 360 days) as is done in [7]. These settings are

compared to the baseline AoA process time which is distributed Tria(180, 360, 720). The null hypothesis for the t-

tests is ��:����� � ��� which corresponds to no significant difference between the baseline and the � �� AoA

triangular distribution variance setting if not rejected and alternative hypothesis ��:����� � � ��� if there is

significant difference.

Table 4: t-test Results for the Variance Reduction of AoA Process Time

AoA Distribution Settings

Tria(180, 360, 720)

(Baseline)

Tria (270, 360,

540)

Tria (315, 360,

450)

Tria (337.5,

360, 405)

Tria (360, 360,

360)

Average Time to

MS-C (Days)

6903.567

6883.159 6867.149 6865.280

6848.717

Standard Deviation

(Days)

1584.258 1591.61 1591.442 1590.301 1576.526

T-Value 0.2528 0.4512 0.4745 0.6828

Conclusion Fail to Reject

H0

Fail to Reject

H0

Fail to Reject

H0

Fail to Reject

H0

It is evident that reducing the variability of the AoA process time does not have an effect on the total time ACAT 1

programs reaches milestone C. Hence any improvement on the AoA process to make it more consistent and standard

would not have a significant effect on the total program completion time.

On the other hand, table 5 presents the results of reducing the mean process time of the AoA from section 3.2 Four

settings are tested specifically, Tria(135, 270, 540), Tria(112.5, 225, 450), Tria(101.25, 202.5, 405), and Tria(90,

180, 360). These settings are selected to illustrate the sensitivity of the length of the AoA processing time from the

baseline scenario to the fastest scenario (Tria(90, 180, 360)). These settings are compared to the baseline AoA

process time which is distributed Tria(180, 360, 720). The null hypothesis for the t-tests is ��:����� � ��� which

corresponds to no significant difference between the baseline and the ��� AoA triangular mean time distribution

setting if not rejected and alternative hypothesis ��:����� � � ��� if there is significant difference.

Table 5: t-test Results for the Mean Reduction of AoA Process Time

AoA Distribution Settings

Tria(180, 360, 720) Tria (135, Tria (112.5, Tria (101.25, Tria (90, 180,

2115

Jalao, Worger, Wirthlin, Colombi, Wu

(Baseline) 270, 540) 225, 450) 202.5, 405) 360)

Average Time to

MS-C (Days)

6903.567 6827.941 6752.989 6732.038 6681.659

Standard Deviation

(Days)

1584.258 1597.607 1578.558 1567.143 1574.53

T-Value 0.9351 1.8732 2.1415 2.7640

Conclusion Fail to Reject

H0

Fail to Reject

H0

Reject H0 Reject H0

It is evident that reducing the AoA process time to at the least Tria(112.5, 225, 450) will result in a failure to reject

that the ��� percentage is similar to the baseline scenario. This implies that reducing the AoA process time to at most

Tria(101.25, 202.5, 405) contributes to a significant change in the total program completion time.

5. Conclusions The analysis of alternatives (AoA) process is a new requirement that all acquisition category (ACAT) level projects

to be completed by the US government are required to undergo. Previously, this process is only required on all

ACAT 1 programs and ACAT 2 and 3 programs with sufficient funding. Hence, program completion is expected to

increase further given that a majority of the programs finish over budget and with a delayed schedule when AoA is

required on all programs. This concept is validated through simulation documented in this paper. By requiring all

programs to undergo an AoA, significant completion delays are observed. Furthermore, this paper provides

quantitative insights on the amount of time that the AoA contributes to the overall DoD end-to-end process time

through Wirthlin [7]’s discrete event system simulation model.

Through additional simulation analysis, it is inferred that a majority of the ACAT 2 programs can be subject to the

AoA without significantly affecting completion time. However, requiring an AoA on ACAT 3 programs has a

significant effect on program completion time. A frequently cited reason for not conducting an AoA is that an AoA

causes a significant delay in the program. The simulation results in this paper should provide strength to the

argument towards providing funds for full AoAs. However, as there is a percentage of programs where the AoA can

cause significant delays in program completion (87.5% and 55% for ACAT II and ACAT III respectively), policy

improvements with regards to selection criteria for programs that must undergo an AoA may be beneficial.

Support for this conclusion can be found in the system engineering case study for the A-10 Thunderbolt II

performed by [20]. From this case study, it can be seen that the correct level of focus is needed for an AoA to be

beneficial. During the development of the A-10 three prototyping studies are performed. These prototyping efforts

are analogous to an AoA. One prototype is a system level test in the form of a fly-off. The other two are subsystem

prototyping of the guns and ammunition for the aircraft. While the system level prototyping provided little

meaningful information, the two subsystem prototyping efforts provide benefits to the DoD in terms of both overall

cost reduction and design improvements. The findings from the case study in conjunction with the simulation results

in this paper indicate that policy applying tailored criteria for ACAT 2 and ACAT 3 AoAs for programs that must

undergo an AoA could be beneficial.

Furthermore, findings from this paper show that any variance reduction improvements or standardizations on the

AoA process would not affect the total program completion time of ACAT 1 programs. However, any

improvements that reduce the processing time of the AoA, or in other words making it more efficient and lean

would significantly reduce program completion time. With the DoD’s requirement to conduct an AoA, an implicit

assumption has been made that the step will shorten the overall lifecycle development. The simulation’s finding

suggests that from a statistical perspective, the DoD must be expeditious during the AoA process, or the value

sought from requiring the AoA decreases. Hence, policy improvements with regards to an AoA should focus on

reducing processing and execution time of the AoA without sacrificing quality in its output.

The use of simulation in this study places some limitations on this research as well as provides opportunities for

further research. One limitation is that the simulation takes a program rather than a portfolio perspective. Previously,

fewer AoAs were performed because of lack of funding. The need to allocate funding to AoAs for ACAT 2 and

ACAT 3 programs may result in more delays across the DoD portfolio because funding may be spread too thinly.

This effect cannot be illustrated by the model if it is present. Further research may be needed to investigate this

2116

Jalao, Worger, Wirthlin, Colombi, Wu

possibility. In addition, AoAs were implemented with the assumption that overall program quality would increase

with the inclusion of an AoA. The simulation does not incorporate the interactions between the AoA and quality.

Further study can be conducted to investigate these interactions or add them to the ERAM simulation.

Acknowledgements This publication is developed under work supported by the Naval Postgraduate School Acquisition Research

Program Assistance Agreement No. N6227112MPZG201 “Enterprise Requirements and Acquisition Model

(ERAM) Analysis and Extension” awarded by the Naval Supply Systems Command (NAVSUP) Fleet Logistics

Center San Diego (NAVSUP FLC San Diego).

References 1. Schwartz,M., 2010, "Defense acquisitions: How DoD acquires weapon systems and recent efforts to reform the

process," DIANE Publishing.

2. Sullivan,M.J., Andrew,C., Bowman,R., Fairbairn,B., Neill,S., Oppenheim,J., Schwenn,R., 2009, "Defense

Acquisitions.Measuring the Value of DOD's Weapon Programs Requires Starting with Realistic Baselines,"

DTIC Document.

3. Arena,M.V., Obaid,Y., Galway L. A., Fox,B., Graser,J.C., Sollinger,J.M., Wu,F., W. Carolyn, 2006, "Impossible

Certainty: Cost Risk Analysis for Air Force Systems," Santa Monica, CA: RAND Corporation.

4. Chairman Joint Chief of Staff (CJCS)., 2012, "CJCS Instruction 3170.01H," CJCS.

5. US 111th Congress, 2009, "Weapon Systems Acquisition Reform Act of 2009," Congress: Washington DC.

6. USD(AT&L), 2008, "Department of Defense Instruction 5000.02," Operation of the Defense Acquisition System.

7. Wirthlin,J.R., 2009, "Identifying enterprise leverage points in Defense Acquisition Program performance," MIT

Dissertation.

8. Georgiadis,D.R., Mazzuchi,T.A., Sarkani,S., 2012, "Using multi criteria decision making in analysis of

alternatives for selection of enabling technology," Systems Engineering. doi: 10.1002/sys.21233.

9. Leach,D., Searle,C., 2011, "Department of Defense Enterprise Requirements and Acquisition Model," Air Force

Institute of Technology Master's Thesis.

10. Montgomery,J., 2012, "Department of Defense Update to the Enterprise Requirements and Acquisition Model,"

Air Force Institute of Technology Master's Thesis.

11. Houston,D., 2012, "Enterprise Requirements and Acquisition Model," Unpublished Manuscript.

12. Cervantes,M.A., Enderton,C., Powers,J.S., 2012, "Rapid Execution of an Analysis of Alternatives for NATO

Special Operations HQ A Smart Defence Approach," Monterey, California. Naval Postgraduate School.

13. Sullivan,M.J., 2012, "Presidential Helicopter Acquisition: Effort Delayed as DOD Adopts New Approach to

Balance Requirements, Costs, and Schedule," GAO.

14. Schank,J.F., 2012, "Analysis of Alternatives: Keys to Success," DTIC Document.

15. TRAC., 2011, "TRADOC Assessment AoA Guidance in DoD 5000 Series," WSARA and In Practice.

Unpublished Manuscript.

16. Stadterman,T.J., Barber,E., Ground,A.P., 2012, "Improving US Army Analysis of Alternatives to Better Address

the Weapon Systems Acquisition Reform Act of 2009," SSCF RESEARCH REPORT.

17. Ford,D.N., Housel,T., Dillard,J.T., 2010, "Integrating System Dynamics Modeling and Knowledge Value Added

for Improved Analysis of Alternatives: A Proof of Concept Study," DTIC Document.

18. Sullivan,M.J., Schwenn,R.E., Bleicher,N.B., Marchesani,S.V., Patton,K.E., Richey,K.A., Russell,A.K.,

Tranquilli,N.A., 2011, "DoD Weapon Systems: Missed Trade-off Opportunities During Requirements

Reviews," GAO.

19. Roper,M., 2010, "Risk Considerations in Pre-Milestone-A Cost Analysis," . Office of the Deputy Assistant

Secretary of the Army for Cost and Economics.

20. Jacques,D.R., Strouble,D.D., 2010, "A-10 Thunderbolt II (Warthog) Systems Engineering Case Study," Air

Force Center for Systems Engineering (AFIT/SY).

2117

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.