Unit VIII Scholarly Activity

This article provides insights into the current state of developmenta l testing (DT) and requirements management in Department of Defense information systems employing Agile development. The authors describe the study methodolog y and provide an overview of Agile development and testing. Insights are described for requirements, detailed planning, test execution, and reporting. This work articulates best practices related to DT and requirements management strategies for programs employing modernized Software Development Life Cycle practices.

DOI: https://doi.org/10.22594/dau.19-819.26.02 Keywords: Development and Operations (DevOps), DoD, Software Development, Automated Testing, Software Development Life Cycle (SDLC)

D EF

EN SE

A CQ

UISITION UNIVER SIT

Y H

IR SC

H R

ES EARCH COMPETIT

IO N

2019

A LU M NI A SS O CIA

TI

O N2nd

DEVELOPMENTAL TEST and REQUIREMENTS: Best Practices of Successful INFORMATION SYSTEMS USING AGILE METHODS

Jeremy D. Kramer and Lt Col Torrey J. Wagner, USAF

 Image designed by Michael Krukowski

130 131Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

Industry software development efforts have used Agile and development and operations (DevOps) methodologies over the last 5 to 15 years. The Department of Defense (DoD) has applied these methodologies to various information system acquisition programs, and current guidance provides a renewed interest in pursuing these methodologies. The National Defense Authorization Act for Fiscal Year 2018 (NDA A, 2017) directs acquisition Program Ma nagement Off ices (PMO) to pursue Agile or iterative sof t- ware development by establishing pilot programs to use “Agile or Iterative Development methods to tailor major software-intensive warfighting sys- tems and defense business systems” (§ 873) and “Software Development Pilot Program Using Agile Best Practices” (§ 874). Similarly, the 2018 Air Force Guidance Memorandum for Rapid Acquisition Activities (A FGM 2018-63-146-01) states that, “Agile software development and [DevOps] is required for all new initiatives unless waived” (Department of the Air Force, 2018, p. 9).

Background This study was motivated by the 2018 NDAA and increased emphasis on

Agile development throughout the DoD; the lead author collected data from April to September 2018 during a rotational assignment within

the Office of the Deputy Assistant Secretary of Defense for Developmenta l Test and Eva luation, DASD(DT&E).

The purpose of the study was to collect insights and best practices from DoD program offices prac-

ticing Agile software development methods, some of which use the DevOps development

and deployment strategy.

The f ive DoD sof tware-intensive sys- tems shown below were selected for

this study, as they have been infor- m a l ly r e c o g n i z e d a s s uc c e s s f u l development programs for their use of Agile methodologies, integrated testing, and/or iterative develop- ment. The authors are not aware of a ny independent rev iew t hat deter mines if t he prog ra ms a re achieving their objectives.

• Joi nt Space Operations Center (JSpOC) Mission System (JMS), Air Force

• D i s t r ibut e d C om mon Gr ou nd S y s t em–Nav y ( D C G S -N ) Increment 2, Navy

• Globa l Combat Suppor t System–Joint (G CSS-J ), Defense Information Systems Agency (DISA)

• Reserve Component Automation System (RCAS), Army

• Cat apu lt /A N TS (At t a ck t he Net work Tool Su it e), Joi nt Improvised-Threat Defeat Organization (JIDO)

Table 1 provides detailed characteristics of the programs that we surveyed and ana lyzed for their Agile development and test approaches. The pro- grams are arranged into two groups: System of Systems (SoS) and Web Applications (Web Apps). The Acquisition Category (ACAT) is either: (a) Automated Information Systems with the Defense Acquisition Executive (IAM), or ( b) DoD Component head (IAC), or designated delegates as the decision authority.

TABLE 1. PROGRAM INFORMATION

Program Description Group Component ACAT

JMS Space command and control situational awareness

SoS Air Force IAM

DCGS-N Increment 2

Tactical gateway for Navy-unique sensor data across the intelligence community

SoS Navy IAC

GCSS-J Joint military integrated logistics through applications and tools

Web Apps DISA IAC

RCAS

Integrated, web-based software solutions to manage mobilization, safety, personnel, and force authorization requirements

Web Apps Army IAM

Catapult/ ANTS

Catapult integrates global intelligence data; ANTS addresses improvised explosive device threats

Web Apps JIDO N/A

The SoS and Web Apps groupings are general in nature. The web application suites (GCSS-J, RCAS, and Catapult/ANTS) consist of one infrastructure system feeding multiple web applications, generally with loose coupling.

132 133Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

Replicating the production environment in a development ecosystem is usu- ally straightforward. On the other hand, SoS (DCGS-N, JMS) are typically more complex than web application suites because they likely depend on tightly coupled, disparate systems that are sensitive to changes. Different vendors may have developed each subsystem, which increases integration challenges due to proprietary information or custom interfaces.

Agile Software Development Agile Software Development Life Cycle (SDLC) methods have emerged

as industry best practices. The Agile methodology emerged in 2001 when 17 leading software developers created the Agile Manifesto to design and share better ways to develop software. Agile va lues can be distilled into four core elements:

• Focusing on small, frequent capability releases

• Valuing working software over comprehensive documentation

• Responding rapidly to changes in operations, technology, and budgets

• Actively involving users throughout development to ensure high operational value

Each of the ma ny existing A g i le methodolog ies (e.g., Scr um, Ex treme P rog r a m m i n g [X P], K a nba n) h a s it s ow n u n ique proc e s se s , t er m s , techniques, and timelines.

DevOps Software Development Dev Ops SDLC met hodolog ies bui ld upon A g i le met hodolog ies by

addressing information technology (IT) operations team deployment activ- ities, resulting in faster deployment to the production environment—daily or weekly instead of monthly or semiannua lly. DevOps works best when the system is in sustainment, when enhancements are smaller, and when a mature user base is established. Increased emphasis is placed on shifting sequential processes into parallel processes and deploying capabilities to the user faster, which entails (Cagle, Kristan, & Rice, 2015):

• Strong use of automation (code, build, infrastructure/archi- tecture, test)

• Continuous development, test, integration, and demonstration of the software throughout the life cycle

• A high level of collaboration and integration between develop- ment, quality assurance, and IT operations

• Static, dynamic, and fuzz testing techniques

• Open source and reusable components, services, and develop- ment tools

A variant of DevOps is Secure DevOps, which emphasizes security through- out the entire process. This variant is known as SecDevOps and DevSecOps, a nd incorporates automated security testing, monitoring, a nd securely handling deployment, typically through hashed Docker containers.

Method A team of DASD(DT&E) staff specialists developed a 38-question sur-

vey for PMO personnel to complete between June and August 2018, which contained questions in these areas, with specific survey questions available upon request:

• Agile development, specifica lly method, management tools, and structure of teams

• Requirements tool usage

• Documents used during actual development

134 135Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

• Test planning, test events, and test implementer (developer, DT&E, operational test and evaluation [OT&E])

• Greatest successes a nd cha llenges to DT&E prog ra m, a nd future goals

• Lessons learned and best practices in the areas of:

° Requirements

° Detailed planning

° Test events and execution

Additiona lly, the tea m inter viewed the 45th a nd 47th Test Squadrons (TS) at Eglin Air Force Base (A FB), FL , to learn more about their Agile developmental testing (DT) efforts. The 45th TS is the principal Air Force organization for strategic air and space, weapon platform, and business systems software DT&E. The 47th TS is the principal Air Force organization that eva luates cyber resiliency of both information- and avionics-based systems. Both belong to the 96th Cyberspace Test Group (F. B. Chavers, personal communication, June 13, 2018).

Analysis The following sections compare and contrast the detail between the

software development methods, requirements, detailed planning, and test events of the five programs studied. Emphasis is placed on automated test- ing, sprint-level testing, and user acceptance testing.

SDLC Method Table 2 identifies each program’s SDLC method, Agile type, tools, and

method of defining requirements.

TABLE 2. PROGRAM SDLC

Program SDLC Method Agile Type Tools Requirements Definition

JMS Scrum/Waterfall Hybrid Jira, self-developed User Story

DCGS-N Increment 2

Scaled Agile Framework (SAFe)

Hybrid Jira User Story

GCSS-J Scrum Agile Rally, Jira User Story

RCAS SAFe, Secure DevOps

Hybrid VersionOne User Story

Catapult/ ANTS Secure DevOps Hybrid Jira User Story

The programs use industry standard Agile SDLC tools, as referenced in the 12th Annual State of Agile Report, which states that 58% of self-reported agencies use Jira, 20% use VersionOne, 9% use CA Agile Central (Rally), and 46% use Excel (respondents could select multiple items) (CollabNet VersionOne, 2018). Within the report, “Respondents were asked whether they would recommend the tool(s) they are using based on their experience,” (p. 15) with 81% recommending VersionOne, 77% recommending Jira, and 72% recommending CA Agile Central (Rally), the top three tools.

Within the GCSS-J program, software developers use Rally for SDLC, and the testers use Jira for user stories and wireframes (detailed diagrams) to help design test cases. Jira provides issue/bug tracking and workflow management.

JMS uses Jira and a self-developed tool for internal tracking of requirements, which are user stories, workflows, and requirements documents. The user stories are derived from workflows or checklists from current operations. Test cases are built over a compilation of user stories that sum to a specific functionality.

136 137Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

VersionOne, used by RCA S, ha s a bu i lt-i n requ i rement s t ra ceabi l it y matrix from capability to feature, to story, to task, to test case, which is extracted and delivered as a part of each release. The user stories contain the accepta nce criteria used to va lidate each stor y during sprint-level testing. Teams also use various levels of workflows and conceptual design specifications based on the need of the specific feature.

One observation by the PMO is the need to fully train personnel in tool use to make maximum use of the capabilities provided by the tools they employ. Without additional system configuration, many of these tools do not work in DoD environments because of security and/or architecture limitations. An example is a defect-tracking system that uses its own server and struc- tured quer y la ng uage database, which requires more tha n a client side installation.

Ag ile method types and scope. Table 3 summa rizes the method, timeframe, and team attributes of the programs studied, which genera lly depended on the complexity a nd size of the project a nd the maturity of the team.

TABLE 3. PROGRAM AGILE SPECIFICS

Program Method Iteration Duration # of Iterations to a Release Sprint Teams

JMS Scrum/Waterfall 1 month, 2 month integration

6 to 20 depending on Service Pack

1 Integrator 4 to 6 Vendors

DCGS-N Increment 2

SAFe 1 month

3 sprints per Capability, 4 per Fleet Capability Release

6 working in parallel, started with 2

GCSS-J Scrum 1 month 4, release capabilities as completed

4 in parallel

RCAS

Hybrid with XP, Kanban and Secure DevOps. SAFe compliant

3 weeks

4 sprints, quarterly releases. 1 week release planning

18 teams across RCAS portfolio 8 to 14 people per team, 2 separate locations

Catapult/ ANTS

Hybrid Agile, Secure DevOps

2 days to 2 weeks

1 month minor, 3 months large

N/A

The Secure Dev Ops PMOs (RCA S a nd Catapu lt/A N TS) stressed t he importance of teamwork and culture to support their efforts, while other PMOs ta lked about the benef its of good communication a nd dedicated

support from different acquisition team members. RCAS and Catapult/ANTS PMOs both remarked on how much time it took to achieve their current DevOps cultures. Both programs started around 2012/2013 and extensively customized their SDLC ecosystems. Both PMOs have delivered ecosystems that allow continuous planning, construction, integration, and delivery.

Catapult/ANTS PMO further noted that it has taken 6 years to transition from waterfa ll to a Secure DevOps environment. Major efforts included bringing in coaches to teach and streamline processes toward Agile (Scrum) development approaches from 2012 through 2014, bringing in automated operating system-level virtualization tools in 2015, and finally achieving a continuous authorization risk management framework pipeline after 2016.

Requirements Gathering Users provide requirements to enable the PMO to build a capability

to meet the requested need. For requirements gathering, user community functiona l representatives send requirements to the development team, with details that vary based on the program’s SDLC approach (Table 2). For GCSS-J, the PMO requirements teams meet with functional requirements g roups, a nd then work w ith the developer to cra f t user stories for the program. Catapult/ANTS PMO receives new requirements in three ways: (a) any user can submit a help desk ticket to address a software defect or new requested functionality; (b) JIDO’s mission IT development engineers can collect requirements from customers in the field; and (c) forward-deployed personnel collect requirements while embedded at the intelligence analysis centers. For example, personnel may see the va lue of a new requirement in an analysis ad hoc report and turn it into an enterprise solution for all analysis centers. The PMO also receives new mission requirements from DoD, JIDO, or other mission-related initiatives; a nd then development engineers work closely with users, Combatant Command staff members, and embedded analysts to vet, identify, and rank new requirements. RCAS PMO has a similar process for gathering requirements from the user, meeting with users on-site for each sprint.

The purpose of the study was to collect insights and best practices from DoD program offices practicing Agile

software development methods, some of which use the DevOps development and deployment strategy.

138 139Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

Detailed Planning All programs use initial iteration-planning meetings, and variations in

their processes are outlined in this section.

For JMS PMO, user stories are sent to the integrator to be approved and executed. User stories that impact multiple requirements can be shared across vendors. However, the cross-coordination of requirements between vendors is a challenge.

RCAS a nd Catapult/A NTS PMOs both employ da ily scrum meetings to manage development through their requirements backlog. Each backlog item has its associated tests identified as “To be built” and “To be executed.” Stories cannot be closed until all tasks are built, all tests passed, and both the product owner and user community accept the solution.

GCSS-J PMO holds weekly and/or biweekly working-level integrated product team (WIPT) meetings with all stakeholders to develop and document test plans for every release. Test dates are included in the integrated master schedule.

DCGS-N PMO assigns testers to each sprint team to participate in scrums, which are daily stand-up meetings. Weekly internal test-planning meetings are also held. External test and evaluation (T&E) WIPT meetings are held with stakeholders.

Test Events and Execution The traditional roles (developer, integrator, and government tester) and

test events are more efficient when reorganized in the Agile environment; however, A g i le a nd Dev Ops br ing new cha l lenges to test events. The developer integ ration tests a re t y pica lly executed by the development team, and System Integration Tests (SIT) are executed by the integrator or

government testers. System Qualification Tests (SQT), or system acceptance tests, are run by government testers. Depending on the circumstance, the roles may be dua l-hatted (i.e., developers may both code and test), or the phases may be merged, as DevOps removes the separate integration step.

Role of the government tester during development sprints. Even when testers are tasked with development duties, they need to ensure that the code developed during the sprint meets the “def inition of done” by thorough automated or manual testing.

With DCGS-N PMO, government testers are assigned to each sprint team and participate in scrums to provide their perspective, working side-by- side with the developer. The 45th TS is reviewing ways to better support development sprints, such as retraining their personnel to better support automated testing or provide specific manual testing solutions.

Catapult/ANTS PMO has encouraged its testers to participate in other parts of the development chain so they can learn test automation, engineering, requirements gathering, and other disciplines.

Role of the developer during government DT. Some programs have separate government DT in the form of SITs, SQTs, or User Acceptance Tests ( UAT), where development contractors prov ide subject mat ter expert support during government DT. DCGS N contractors also provide user training, technica l support, maintenance, and simulation support. JMS vendors are present during 45th TS government testing to speed up defect turnaround, so that when a system defect is found, the vendor can immediately understand what is happening and start to develop a solution.

Test t y pes . T he developer a nd gover n ment t est ers per for m t he following levels of test: automated testing, sprint-level testing, and user acceptance testing. This is not an all-inclusive list.

Auto m ate d s of t w a re t e s t in g . Aut om at e d s of t w a re t e s t i n g i s conducted as required, generally at each code compilation. Programs use, or intend to use, automated tools as much as possible because of the recognized time and cost savings. However, the hardware and software ecosystem can make the return on investment (ROI) difficult.

Programs with a DevOps pipeline use robust automated tools because they a re a lso testing for integ ration. RCAS a nd Catapult/A NTS PMOs both note that no single tool works best, and that automation frameworks are a collection of tools used to support automation across the Agile solution delivery life cycle.

140 141Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

Catapult/ANTS PMO has a continuous integration and deployment pipeline that uses Jira, Jenkins, and Docker tools. Specifically, the bulleted list of testing/quality gates is performed in the following order. If these tests are not passed, then code deployment is halted and left in the queue for examination (L. T. Garciga & D. Lederman, personal communication, July 2, 2018):

• Unit Tests (Apache Maven, JaCoCo, SonarQube)

° Code Coverage Percentage

• Static Code Analysis (SonarQube, Sonatype)

° Code Quality

° Refactor Opportunities

° Possible Injection Errors

° Vulnerability Scan

• Exchange-to-Exchange Tests (Selenium, Protractor)

° Automated Integration Tests

° Ensure Application is Operational

° Stress Testing

• Penetration Testing (Open Web Application Security Project Zed Attack Proxy)

Catapu lt/A NTS PMO f ur ther pursues a continuous risk ma na gement approach by providing the information assurance team with direct access to live tools including SonarQube and Twistlock. These tools help the PMO track and verify tasks in its plans of action and milestones, and provide a n automated lockdown that addresses 240 of 242 Securit y Technica l Implementation Guide (STIG) ma ndated controls (L . T. Ga rciga & D. Lederma n, persona l communication, July 2, 2018). The rema ining two STIG controls must be verified manually.

RCAS PMO uses a broad variety of tools in its Secure DevOps framework, supporting a variety of heterogeneous architectures. Key tools for the RCAS por tfolio include CodedUI, Specf low, Cucumber, Chef, A r tifactor y, C#, VisualStudio, PowerShell, Selenium, NUnit, Oracle, and Moq.

DCGS-N uses JMeter for functiona l testing and Sta llion for automated message and data file injection, and the other programs use similar tools and static code analysis.

Sprint-level testing. Certain types of testing are more appropriate for the end of the sprint. The Space and Naval Warfare Systems Command (SPAWAR) performed additional testing during integration for JMS and DCGS-N. DCGS-N schedules cumulative functiona l testing before each Capability Drop and before each Fleet Capability Release. SPAWAR executes an integration test for each Fleet Capability Release before the system goes through a developmental testing/operational testing (DT/OT) cycle.

GCSS-J PMO performs developer testing at the sprint level. When sprint drops are integrated, the government does release testing, UATs, and system acceptance tests, which mark the forma l end of government acceptance testing for the release.

RCAS PMO performs all testing within each sprint so that each backlog item is a potentially shippable microrelease. This testing includes functional and nonfunctional testing; unit and integration testing; performance testing; vulnerability testing; acceptance testing; white-box and black-box testing; and automated unit, smoke, regression, and vulnerability testing.

User acceptance testing. Government UAT has different meanings among the programs, but usua lly it involves the first use by actua l users in a government-led event. Release teams genera lly consist of the PMO, developer, government T&E, and user community personnel. Oftentimes, members of these development teams are drawn from organizations outside the PMO. The user community is a key component of the release team for programs using Agile development methodologies. Government UAT is genera lly performed at the end of a release, but earlier government test involvement during daily, sprint, and integration testing can add value to the effort.

When government testers collaborate with the development team, the sharing of test plans, test cases, test

data, and defects enhances testing and future software development.

142 143Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

G CSS -J ’s test tea m receives operationa l m ission t h rea ds f rom ea ch command and incorporates them into testing. A team comprising Joint Interoperability Test Command (JITC) OT&E personnel and DISA DT&E personnel conducts an integrated SQT at the end of a release. If successful, t his approach avoids a dedicated relea se OT&E . The DISA test tea m considers this working relationship with JITC and the stakeholders to be its greatest success.

At Va ndenberg A FB, CA , JMS PMO hosts a progra m insta nce that ca n gain feedback from operators without their having to leave home stations. Acquiring dedicated operators for a dedicated test event is difficult, but this approach allows early user feedback. Additionally, JMS PMO tasks the 45th TS to provide its government DT, and the PMO prefers a smaller team dedicated to its effort, rather than the larger team that is shared among other 45th TS projects. For its current service pack, JMS PMO is working to involve the 45th TS more during contractor and integrator testing. The PMO is also involved with the Air Force OT&E Center, which provided an OT&E perspective earlier in the project.

For RCAS PMO, the user community must accept new features and releases during sprints for them to be considered complete. Catapult/ANTS PMO employs users as a test group in UATs, and further monitors user feedback a f t er deploy ment t h rou g h a ser ies of en ga gement s t o mon it or r i sk s associated with community adoptions and mission effectiveness.

Metrics. Programs monitor their testing and development activities using manua l and automated tools to track defects and test completion metrics, depending on the level of complexity and maturity of their solutions.

RCAS PMO tracks the productivity, capacity, and efficiency of their Agile Solution Factor y (ASF) using stor y point a na lysis, which is conducted after each RCAS quarterly release to measure performance metrics across multiple levels from scrum teams to the enterprise level. The performance a na lysi s del ivers t ra n spa renc y a nd pred ict abi l it y of t he A SF t o t he PMO. Other performa nce a na lysis approaches used in pa ra llel include sustainment analysis, Agile earned value management, story point defect a na lysis to mea sure relea se qua lit y, deliver y velocit y, automation ROI analysis, and help desk ticket analysis. A combination of these approaches allows RCAS PMO to continuously optimize the performance and efficiency of their program.

JMS PMO tracks findings using the deficiency report processes, and JIDO uses tools such a s JaCoCo to provide test ca se code coverage metrics. DevOps organizations monitor their processes with automated tools.

Best Practices The observed best practices are specific to each program; however, they

can be generalized to provide a guide to improve software acquisition for other programs.

Develop a Tailored Approach with Continuous Delivery Each PMO selected an SDLC methodology that best met its needs; no

clea r linkage emerged between the progra m groupings a nd their SDLC met hod. Table 4 shows t he prog ra m g roups a nd SDLCs used for each program.

TABLE 4. PROGRAM GROUP AND SDLC

Program Group SDLC

JMS SoS Scrum

DCGS-N SoS SAFe

GCSS-J Web Applications Scrum

RCAS Web Applications SAFe, Secure DevOps

Catapult/ANTS Web Applications Secure DevOps

Drawing from the discussion in the Background section, the web application suites (GCSS-J, RCAS, and Catapult/ANTS) are more suitable for DevOps methodologies, as they are loosely coupled, allowing the development and deployment of small pieces over time. SoS complexity (DCGS-N, JMS) does not preclude the use of DevOps, but makes implementation difficult—both PMOs noted difficulty integrating input from multiple vendors. Deployment timing is also a factor; DCGS-N may deploy to a ship only once a year, which limits the ROI from a continuous deployment approach.

PMOs should also consider tailored approaches based on their requirements and time frames. Catapult/ANTS PMO uses different Agile methodologies based on requirements cha racteristics. For well-def ined requirements with minimal user input, the PMO uses Agile Kanban SDLC methodologies to deploy a solution in as little as 2 days. For vague requirements or those t hat i nvolve subst a nt ia l user i ntera ct ion , a t ra d it iona l A g i le Scr u m approach is used. Wagner and Ford (2018), in their journal article entitled, DoD Applications of Agile Software Development Methods, a lso reach a

144 145Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

comparable conclusion. “No single methodology is suited for all software projec t s,” a nd “ Developer s, m a n a gers, a nd s t a keholder s (especia l ly operational stakeholders) should jointly choose an Agile method based on project type, project size, the amount of documentation required, project risk, and the experience level of the development team” (Wagner & Ford, 2018, p. 52).

Multiple PMOs noted the merits of continuous delivery models, and JMS and GCSS-J PMOs are working to shift their processes toward continuous delivery models. JMS PMO specifically sees an advantage toward smaller pieces of capability that are effective by the user community. PMOs such as RCAS and Catapult/ANTS have implemented continuous delivery over the last 5 years in their Secure DevOps implementation. The Defense Science Board makes a similar recommendation that PMOs pursue a “Continuous It erat ive Development ” approa ch , w it h a n emph a si s on ident i f y i n g Minimum Viable Product approaches and delegating acquisition authority to the PMO (DoD, 2018).

Emphasize Obtaining Quality Feedback on Requirements and Solutions

Agile methodologies stress the day-to-day involvement of the product owner a nd/or user community. Some PMOs have embedded traditiona l requirements development organization representatives within their PMO to facilitate rapid exchange of ideas, clarification of requirements, and real- time feedback.

Several PMOs have active requirements development or gathering teams w it h i n t he PMO. The PMOs use t hese tea ms to ref i ne requ i rement s provided by the functional sponsor and/or users. The team also collects new requirements once the program has transitioned into sustainment, although DevOps programs may also practice this approach during development. Best practices are to delegate requirements development and refinement down to the PMO level and maintain a constant connection between the developers and user community.

Catapult/ANTS PMO has dedicated development engineers embedded at user facilities with intelligence analysts at the point of need. For example, personnel may see the va lue of a new requirement in an ana lysis ad hoc report and turn it into an enterprise solution for a ll ana lysis centers. In another example from Catapult/ANTS, the PMO selected an open-source graph database supported by the community, but after implementation, the open-source community had moved in a different direction with a different database. The PMO readjusted and supported the new database quickly rather than implementing a stale code release.

Invite Developers and Users to Participate in Government Testing

The key for successful DT is to communicate defect information back to the developer as quickly as possible. Although this objective could also be accomplished through a shorter reporting process, having the developer present when the errors occur has been shown to be one of the best methods.

Development tea m members provide a useful resource to the test tea m during and after government testing. Their knowledge of system design can be compared to user expectations during testing. Developer presence during government DT provides critical feedback to support efficient and effective defect resolution. Additionally, user involvement helps find different types of errors because the users may use the system differently than the developer and/or tester imagined. Government test organizations can provide test jurisdiction with clearly defined processes and procedures in place to keep government DT independent. When government testers collaborate with the development team, the sharing of test plans, test cases, test data, and defects enhances testing and future software development.

In some cases, users cannot participate in a single test event. JMS PMO provides access to a program instance at the users’ home stations since obtaining time from dedicated operators for the test events can be difficult. This approach has consistently provided valuable feedback.

146 147Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

The Return on Investment for Automated Testing Depends on the Program

Several PMOs recommended pursuing automated test tools; however, both RCAS and Catapult/ANTS PMOs note that no single tool works best for automation tools. Automated testing involves a significant investment in automated software testing tools and training the workforce to use them effectively. Over time, automated sof twa re testing ca n reduce the time spent on repetitive regression testing, which is essentia l to achieve the flow that makes Agile software development effective. Although automated software testing has great benefits, PMOs will likely still need some degree of manual testing.

Each PMO team selects the appropriate tools for its development pipelines a nd reeva luates as new technolog y becomes ava ilable. A ga in, progra m length and type may impact the ROI.

Specifically, the Catapult/ANTS team has internal processes for validating their automated testing, which has resulted in changes to the automation tools t hat t he tea m uses. They f ur t her mentioned t hat t hey i n itia l ly performed manual and automated testing in parallel to build confidence in the automated tools, and noted that the automated tests hastened the process. The program has reached the point where even most user acceptance testing is automated, except for complex tasks. Portions of their user acceptance testing a re executed either by recording user system interactions a nd replaying them, or by using a program to click through tasks.

Government DT can provide a similar process to guide and oversee auto- mated testing. Government testers can guide automated testing by selecting tools, processes such as design of experiments or combinatoria l testing, gates for the testing pipeline, and specify the extent of testing when config- uring each tool. Once the pipeline is set up, government testers can oversee the process as the product moves through deployment gates to determine whether any issues arise.

Transition to Agile is Not a Quick and Easy Process The two DevOps programs, RCAS and Catapult/ANTS, both developed

their processes over a 3- to 5-yea r period. W hen considering Catapult/ A NTS, t he J IDO Chief Technolog y Of f icer rema rked on t he need for retra ining testers, a nd JIDO’s continued move to Secure DevOps w i ll require “Another 30 percent change in the skillset of the workforce” (Miller, 2018, para. 20). The 45th TS is also investigating focusing their workforce by either retra ining its personnel to better suppor t automated testing, or providing specif ic ma nua l testing solutions. The RCA S PMO noted that the transformation to Agile at an enterprise scale requires a holistic, structured approach across three perspectives: (a) organization and people, (b) process, governance, and method, and (c) tools. Cultural transformations are difficult and require leadership driving the transformation from both the government and the system integrator.

For the RCAS program, both the government and the system integrator supported the continual evolution and transformation of their ASF (O’Brian & Green-Mack, 2018). The system integrator had a structured approach and training to transform to Agile, implement an ASF, and scale the ASF over time across the three perspectives noted previously. The RCAS PMO worked with the system integrator to adapt and enhance these structured approaches to integ rate into bot h t he RCA S PMO a nd RCA S mission space. RCAS PMO noted the benefits of having the system integrator host RCAS development and test environments, which included ma ximizing A SF per forma nce, minimizing costs, increa sing product qua lit y, a nd cybersecurity. The system integrator can quickly integrate new tools within its cloud-based development test environments to drive better performance.

Initial efforts to implement this recommendation have already started. The Defense Acquisition University has undertaken an effort to create an Agile progra m ma nagement course designed to introduce Agile development precepts to pilot programs. This effort is not individua l training, but is intended to instill a cultural change across the program management team

Agile and DevOps software development takes time and effort to implement, but the ROI from implementing automated

procedures, requirements gathering teams, and better collaboration can be of great benefit to most software development programs.

148 149Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

as a group. This course is being developed to fulfill requirements of the 2018 NDAA (§ 891), which will provide training on “agile or iterative development methods to provide training for personnel implementing and supporting the pilot programs required by Sections 873 and 874 of this Act” (NDA A for Fiscal Year 2018, 2017).

Conclusions The DoD continues to pursue Agile and DevOps software development

methodologies, based on strategic guidance, and the potential for cost and schedule efficiencies. Although not all program types may benefit from these development techniques, the paradigm is shifting, and program managers and testers need to be aware of the circumstances where Agile or DevOps methods could benefit their programs. This study surveyed five acquisition programs in the areas of Agile development, requirements gathering, test pla nning, a nd test execution. The a na lysis summa rizes their Agile a nd DevOps SDLC methodologies and the sof tware development strategies. Each program uses unique management software, such as Jira, to track and manage software development. Requirements gathering and test event detailed planning are explored, and the specific test events and executions are covered with an emphasis on what is different during Agile and DevOps software development. The final section distills the findings into lessons learned and best practices. Agile and DevOps software development takes time and effort to implement, but the ROI from implementing automated procedures, requirements-gathering teams, and better collaboration can be of great benefit to most software development programs.

References

Cagle, R., Kristan, M., & Rice, T. (2015). DevOps for federal acquisition. Retrieved from https://www.mitre.org/sites/default/files/publications/15-2842-DevOps-for- Federal-Aquisition.pdf

CollabNet VersionOne. (2018, April 9). 12th annual state of Agile report [Video file]. Retrieved from https://explore.versionone.com/state-of-agile/versionone-12th- annual-state-of-agile-report

Department of the Air Force. (2018). Air Force guidance memorandum for rapid acquisition activities (AFGM 2018-63-146-01). Washington, DC: Office of the Secretary of the Air Force (Acquisition).

Department of Defense. (2018). Final report of the Defense Science Board task force on the design and acquisition of software for defense systems. Retrieved from http://www.dtic.mil/dtic/tr/fulltext/u2/1048883.pdf

Miller, J. (2018, July 27). Emerging threat to warfighters required new thinking by DoD’s JIDO. Federal News Network, Ask the CIO. Retrieved from https:// federalnewsradio.com/ask-the-cio/2018/07/emerging-threat-to-warfighters- required-new-thinking-by-dods-jido/

National Defense Authorization Act for Fiscal Year 2018, 10 U.S.C., Pub. L. 115-91 § 873-891 (2017).

O’Brian, S. P., & Green-Mack, A. D. (2018, April 9). Taking Agile all the way. Army ALT Magazine. Science and Technology. Retrieved from https://asc.army.mil/web/ news-alt-amj18-taking-agile-all-the-way/

Wagner, L. C., & Ford, T. (2018, December). DoD applications of Agile software development methods. Journal of DoD Research and Engineering, 1(3), 6–27. Retrieved from https://www.dtic.mil/DTICOnline/ home.search?q=AD1064303&utm_medium=jdre_homepage&utm_ source=dodtechipedia&utm_campaign=jdredec2018_issue3&utm_ term=jdrearticleCY-18-9#

Acknowledgments

The authors are grateful for significant contributions from DASD(DT&E) staff members Chris DeLuca, Leland Hewitt, and Mark Price. Dr. Ferhan Kilical and Susan Gotta provided substantive input to compilation of the manuscript topics and content.

150 151Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150 Defense ARJ, April 2019, Vol. 26 No. 2 : 128-150

Developmental Test and Requirements: Best Practices of Successful Information Systems https://www.dau.mil April 2019

Author Biographies

Mr. Jeremy D. Kramer is a n operations resea rch a na lyst for A ir Mobilit y Comma nd Test a nd E va luat ion Squa d ron , Joi nt Ba se McGu i re -Di x- Lakehurst, New Jersey. Mr. Kramer holds a BS in Mathematics f rom Sa n Diego State Universit y a nd a n MS in Operations

Resea rch f rom Nor t h Ca rolina State Universit y.

(E-mail address: [email protected])

Lt Col Torrey J. Wagner, USAF, is cur rent ly a facu lt y member at t he A ir Force Instit ute of Tech nolog y (A FIT). Lt Col Wa g ner wa s com m issioned i n 2000 from the University of Minnesota, Minnesota Air Force ROTC, w it h a B S i n Elect r ica l En g i neer i n g. H is prev ious a s s i g n m e n t s i n c l u d e d A i r Fo r c e R e s e a r c h L a b o r a t o r y (A F R L) se c t ion ch ief; A F R L br a nch ch ief, A F R L deput y division chief, and executive officer to the AFRL Commander; director of operations for a 60-member eng ineering a na lysis squadron; a nd a s t he Com ma nder of a n 80 -person applied eng ineering squadron. Lt Col Torrey holds g raduate deg rees

f rom Loug hboroug h Universit y, U K , a nd A FIT.

(E-mail: [email protected])

This content is in the Public Domain.