Short Answer and Discussion (400 words total)

Astrategy for software testing provides a road map that describes the stepsto be conducted as part of testing, when these steps are planned and thenundertaken, and how much effort, time, and resources will be required. Therefore, any testing strategy must incorporate test planning, test case design, test execution, and resultant data collection and evaluation.

A software testing strategy should be flexible enough to promote a customized testing approach. At the same time, it must be rigid enough to encourage reason- able planning and management tracking as the project progresses. Shooman [Sho83] discusses these issues:

In many ways, testing is an individualistic process, and the number of different types

of tests varies as much as the different development approaches. For many years, our

449

C H A P T E R

17SOFTWARE TESTINGSTRATEGIES

What is it? Software is tested to uncover errors that were made inad- vertently as it was designed and constructed. But how do you conduct

the tests? Should you develop a formal plan for your tests? Should you test the entire program as a whole or run tests only on a small part of it? Should you rerun tests you’ve already conducted as you add new components to a large system? When should you involve the customer? These and many other questions are answered when you develop a software testing strategy.

Who does it? A strategy for software testing is developed by the project manager, software engineers, and testing specialists.

Why is it important? Testing often accounts for more project effort than any other software engi- neering action. If it is conducted haphazardly, time is wasted, unnecessary effort is expended, and even worse, errors sneak through undetected. It would therefore seem reasonable to establish a systematic strategy for testing software.

What are the steps? Testing begins “in the small” and progresses “to the large.” By this I mean

Q U I C K L O O K

that early testing focuses on a single component or a small group of related components and applies tests to uncover errors in the data and processing logic that have been encapsulated by the component(s). After components are tested they must be integrated until the complete system is constructed. At this point, a series of high-order tests are executed to uncover errors in meeting customer requirements. As errors are uncovered, they must be diagnosed and cor- rected using a process that is called debugging.

What is the work product? A Test Specification documents the software team’s approach to test- ing by defining a plan that describes an overall strategy and a procedure that defines specific testing steps and the types of tests that will be conducted.

How do I ensure that I’ve done it right? By reviewing the Test Specification prior to testing, you can assess the completeness of test cases and testing tasks. An effective test plan and pro- cedure will lead to the orderly construction of the software and the discovery of errors at each stage in the construction process.

K E Y C O N C E P T S alpha test . . . . . .469 beta test . . . . . . .469 class testing . . . . .466 configuration review . . . . . . . .468 debugging . . . . . .473 deployment testing . . . . . . . . .472 independent test group . . . . . .452

pre75977_ch17.qxd 11/27/08 6:09 PM Page 449

450 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

integration testing . . . . . . . . .459 regression testing . . . . . . . . .467 system testing . . .470 unit testing . . . . .456 validation testing . . . . . . . . .467 V&V . . . . . . . . . .450

only defense against programming errors was careful design and the native intelligence

of the programmer. We are now in an era in which modern design techniques [and tech-

nical reviews] are helping us to reduce the number of initial errors that are inherent in the

code. Similarly, different test methods are beginning to cluster themselves into several

distinct approaches and philosophies.

These “approaches and philosophies” are what I call strategy—the topic to be pre- sented in this chapter. In Chapters 18 through 20, the testing methods and tech- niques that implement the strategy are presented.

1 7 . 1 A S T R AT E G I C A P P R O A C H T O S O F T WA R E T E S T I N G Testing is a set of activities that can be planned in advance and conducted system- atically. For this reason a template for software testing—a set of steps into which you can place specific test case design techniques and testing methods—should be defined for the software process.

A number of software testing strategies have been proposed in the literature. All provide you with a template for testing and all have the following generic characteristics:

• To perform effective testing, you should conduct effective technical reviews (Chapter 15). By doing this, many errors will be eliminated before testing commences.

• Testing begins at the component level and works “outward” toward the integration of the entire computer-based system.

• Different testing techniques are appropriate for different software engi- neering approaches and at different points in time.

• Testing is conducted by the developer of the software and (for large projects) an independent test group.

• Testing and debugging are different activities, but debugging must be accom- modated in any testing strategy.

A strategy for software testing must accommodate low-level tests that are neces- sary to verify that a small source code segment has been correctly implemented as well as high-level tests that validate major system functions against customer requirements. A strategy should provide guidance for the practitioner and a set of milestones for the manager. Because the steps of the test strategy occur at a time when deadline pressure begins to rise, progress must be measurable and problems should surface as early as possible.

17.1.1 Verification and Validation

Software testing is one element of a broader topic that is often referred to as verifi- cation and validation (V&V). Verification refers to the set of tasks that ensure that

WebRef Useful resources for software testing can be found at www.mtsu .edu/~storm/.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 450

software correctly implements a specific function. Validation refers to a different set of tasks that ensure that the software that has been built is traceable to customer requirements. Boehm [Boe81] states this another way:

Verification: “Are we building the product right?”

Validation: “Are we building the right product?”

The definition of V&V encompasses many software quality assurance activities (Chapter 16).1

Verification and validation includes a wide array of SQA activities: technical reviews, quality and configuration audits, performance monitoring, simulation, feasi- bility study, documentation review, database review, algorithm analysis, development testing, usability testing, qualification testing, acceptance testing, and installation test- ing. Although testing plays an extremely important role in V&V, many other activities are also necessary.

Testing does provide the last bastion from which quality can be assessed and, more pragmatically, errors can be uncovered. But testing should not be viewed as a safety net. As they say, “You can’t test in quality. If it’s not there before you begin test- ing, it won’t be there when you’re finished testing.” Quality is incorporated into soft- ware throughout the process of software engineering. Proper application of methods and tools, effective technical reviews, and solid management and measurement all lead to quality that is confirmed during testing.

Miller [Mil77] relates software testing to quality assurance by stating that “the underlying motivation of program testing is to affirm software quality with methods that can be economically and effectively applied to both large-scale and small-scale systems.”

17.1.2 Organizing for Software Testing

For every software project, there is an inherent conflict of interest that occurs as testing begins. The people who have built the software are now asked to test the soft- ware. This seems harmless in itself; after all, who knows the program better than its developers? Unfortunately, these same developers have a vested interest in demon- strating that the program is error-free, that it works according to customer require- ments, and that it will be completed on schedule and within budget. Each of these interests mitigate against thorough testing.

From a psychological point of view, software analysis and design (along with coding) are constructive tasks. The software engineer analyzes, models, and then creates a computer program and its documentation. Like any builder, the software

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 451

uote:

“Testing is the unavoidable part of any responsible effort to develop a software system.”

William Howden

Don’t get sloppy and view testing as a “safety net” that will catch all errors that occurred because of weak software engi- neering practices. It won’t. Stress quality and error detection throughout the software process.

1 It should be noted that there is a strong divergence of opinion about what types of testing consti- tute “validation.” Some people believe that all testing is verification and that validation is conducted when requirements are reviewed and approved, and later, by the user when the system is opera- tional. Other people view unit and integration testing (Sections 17.3.1 and 17.3.2) as verification and higher-order testing (Sections 17.6 and 17.7) as validation.

uote:

“Optimism is the occupational hazard of programming; testing is the treatment.”

Kent Beck

pre75977_ch17.qxd 11/27/08 6:09 PM Page 451

engineer is proud of the edifice that has been built and looks askance at anyone who attempts to tear it down. When testing commences, there is a subtle, yet def- inite, attempt to “break” the thing that the software engineer has built. From the point of view of the builder, testing can be considered to be (psychologically) destructive. So the builder treads lightly, designing and executing tests that will demonstrate that the program works, rather than uncovering errors. Unfortunately, errors will be present. And, if the software engineer doesn’t find them, the cus- tomer will!

There are often a number of misconceptions that you might infer erroneously from the preceding discussion: (1) that the developer of software should do no test- ing at all, (2) that the software should be “tossed over the wall” to strangers who will test it mercilessly, (3) that testers get involved with the project only when the testing steps are about to begin. Each of these statements is incorrect.

The software developer is always responsible for testing the individual units (components) of the program, ensuring that each performs the function or exhibits the behavior for which it was designed. In many cases, the developer also conducts integration testing—a testing step that leads to the construction (and test) of the complete software architecture. Only after the software architecture is complete does an independent test group become involved.

The role of an independent test group (ITG) is to remove the inherent problems associated with letting the builder test the thing that has been built. Independent testing removes the conflict of interest that may otherwise be present. After all, ITG personnel are paid to find errors.

However, you don’t turn the program over to ITG and walk away. The developer and the ITG work closely throughout a software project to ensure that thorough tests will be conducted. While testing is conducted, the developer must be available to correct errors that are uncovered.

The ITG is part of the software development project team in the sense that it becomes involved during analysis and design and stays involved (planning and spec- ifying test procedures) throughout a large project. However, in many cases the ITG reports to the software quality assurance organization, thereby achieving a degree of independence that might not be possible if it were a part of the software engi- neering organization.

17.1.3 Software Testing Strategy—The Big Picture

The software process may be viewed as the spiral illustrated in Figure 17.1. Initially, system engineering defines the role of software and leads to software requirements analysis, where the information domain, function, behavior, performance, con- straints, and validation criteria for software are established. Moving inward along the spiral, you come to design and finally to coding. To develop computer software, you spiral inward (counterclockwise) along streamlines that decrease the level of abstraction on each turn.

452 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

An independent test group does not have the “conflict of interest” that builders of the software might experience.

uote:

“The first mistake that people make is thinking that the testing team is responsible for assuring quality.”

Brian Marick

pre75977_ch17.qxd 11/27/08 6:09 PM Page 452

A strategy for software testing may also be viewed in the context of the spiral (Figure 17.1). Unit testing begins at the vortex of the spiral and concentrates on each unit (e.g., component, class, or WebApp content object) of the software as imple- mented in source code. Testing progresses by moving outward along the spiral to integration testing, where the focus is on design and the construction of the software architecture. Taking another turn outward on the spiral, you encounter validation testing, where requirements established as part of requirements modeling are vali- dated against the software that has been constructed. Finally, you arrive at system testing, where the software and other system elements are tested as a whole. To test computer software, you spiral out in a clockwise direction along streamlines that broaden the scope of testing with each turn.

Considering the process from a procedural point of view, testing within the con- text of software engineering is actually a series of four steps that are implemented sequentially. The steps are shown in Figure 17.2. Initially, tests focus on each

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 453

System testing Validation testing Integration testing

Unit testing

Code

Design Requirements

System engineering

FIGURE 17.1 Testing strategy

Unit testCode

Design

Requirements

Testing “direction”

Integration test

High-order tests

FIGURE 17.2 Software testing steps

What is the overall

strategy for software testing?

?

WebRef Useful resources for software testers can be found at www .SQAtester.com.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 453

component individually, ensuring that it functions properly as a unit. Hence, the name unit testing. Unit testing makes heavy use of testing techniques that exercise specific paths in a component’s control structure to ensure complete coverage and maximum error detection. Next, components must be assembled or integrated to form the complete software package. Integration testing addresses the issues associ- ated with the dual problems of verification and program construction. Test case design techniques that focus on inputs and outputs are more prevalent during integration, although techniques that exercise specific program paths may be used to ensure coverage of major control paths. After the software has been integrated (constructed), a set of high-order tests is conducted. Validation criteria (estab- lished during requirements analysis) must be evaluated. Validation testing provides final assurance that software meets all informational, functional, behavioral, and performance requirements.

The last high-order testing step falls outside the boundary of software engineer- ing and into the broader context of computer system engineering. Software, once validated, must be combined with other system elements (e.g., hardware, people, databases). System testing verifies that all elements mesh properly and that overall system function/performance is achieved.

454 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

The scene: Doug Miller’s office, as component-level design continues and

construction of certain components begins.

The players: Doug Miller, software engineering manager, Vinod, Jamie, Ed, and Shakira—members of the SafeHome software engineering team.

The conversation:

Doug: It seems to me that we haven’t spent enough time talking about testing.

Vinod: True, but we’ve all been just a little busy. And besides, we have been thinking about it . . . in fact, more than thinking.

Doug (smiling): I know . . . we’re all overloaded, but we’ve still got to think down the line.

Shakira: I like the idea of designing unit tests before I begin coding any of my components, so that’s what I’ve been trying to do. I have a pretty big file of tests to run once code for my components is complete.

Doug: That’s an Extreme Programming [an agile software development process, see Chapter 3] concept, no?

Ed: It is. Even though we’re not using Extreme Programming per se, we decided that it’d be a good idea to design unit tests before we build the component—the design gives us all of the information we need.

Jamie: I’ve been doing the same thing.

Vinod: And I’ve taken on the role of the integrator, so every time one of the guys passes a component to me, I’ll integrate it and run a series of regression tests on the partially integrated program. I’ve been working to design a set of appropriate tests for each function in the system.

Doug (to Vinod): How often will you run the tests?

Vinod: Every day . . . until the system is integrated . . . well, I mean until the software increment we plan to deliver is integrated.

Doug: You guys are way ahead of me!

Vinod (laughing): Anticipation is everything in the software biz, Boss.

SAFEHOME

Preparing for Testing

pre75977_ch17.qxd 11/27/08 6:09 PM Page 454

17.1.4 Criteria for Completion of Testing

A classic question arises every time software testing is discussed: “When are we done testing—how do we know that we’ve tested enough?” Sadly, there is no defin- itive answer to this question, but there are a few pragmatic responses and early attempts at empirical guidance.

One response to the question is: “You’re never done testing; the burden simply shifts from you (the software engineer) to the end user.” Every time the user executes a computer program, the program is being tested. This sobering fact underlines the importance of other software quality assurance activities. Another response (some- what cynical but nonetheless accurate) is: “You’re done testing when you run out of time or you run out of money.”

Although few practitioners would argue with these responses, you need more rig- orous criteria for determining when sufficient testing has been conducted. The cleanroom software engineering approach (Chapter 21) suggests statistical use tech- niques [Kel00] that execute a series of tests derived from a statistical sample of all possible program executions by all users from a targeted population. Others (e.g., [Sin99]) advocate using statistical modeling and software reliability theory to predict the completeness of testing.

By collecting metrics during software testing and making use of existing software reliability models, it is possible to develop meaningful guidelines for answering the question: “When are we done testing?” There is little debate that further work remains to be done before quantitative rules for testing can be established, but the empirical approaches that currently exist are considerably better than raw intuition.

1 7 . 2 S T R AT E G I C I S S U E S Later in this chapter, I present a systematic strategy for software testing. But even the best strategy will fail if a series of overriding issues are not addressed. Tom Gilb [Gil95] argues that a software testing strategy will succeed when software testers:

Specify product requirements in a quantifiable manner long before testing com- mences. Although the overriding objective of testing is to find errors, a good testing strategy also assesses other quality characteristics such as portability, maintain- ability, and usability (Chapter 14). These should be specified in a way that is meas- urable so that testing results are unambiguous.

State testing objectives explicitly. The specific objectives of testing should be stated in measurable terms. For example, test effectiveness, test coverage, mean- time-to-failure, the cost to find and fix defects, remaining defect density or fre- quency of occurrence, and test work-hours should be stated within the test plan.

Understand the users of the software and develop a profile for each user category. Use cases that describe the interaction scenario for each class of user can reduce overall testing effort by focusing testing on actual use of the product.

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 455

When are we finished

testing? ?

WebRef A comprehensive glossary of testing terms can be found at www .testingstandards .co.uk/living_ glossary.htm.

What guidelines

lead to a successful software testing strategy?

?

WebRef An excellent list of testing resources can be found at www .io.com/~wazmo/ qa/.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 455

Develop a testing plan that emphasizes “rapid cycle testing.” Gilb [Gil95] recom- mends that a software team “learn to test in rapid cycles (2 percent of project effort) of customer-useful, at least field ‘trialable,’ increments of functionality and/or quality improvement.” The feedback generated from these rapid cycle tests can be used to control quality levels and the corresponding test strategies.

Build “robust” software that is designed to test itself. Software should be designed in a manner that uses antibugging (Section 17.3.1) techniques. That is, software should be capable of diagnosing certain classes of errors. In addition, the design should accommodate automated testing and regression testing.

Use effective technical reviews as a filter prior to testing. Technical reviews (Chapter 15) can be as effective as testing in uncovering errors. For this reason, reviews can reduce the amount of testing effort that is required to produce high- quality software.

Conduct technical reviews to assess the test strategy and test cases themselves. Technical reviews can uncover inconsistencies, omissions, and outright errors in the testing approach. This saves time and also improves product quality.

Develop a continuous improvement approach for the testing process. The test strat- egy should be measured. The metrics collected during testing should be used as part of a statistical process control approach for software testing.

1 7 . 3 T E S T S T R AT E G I E S F O R C O N V E N T I O N A L S O F T WA R E 2

There are many strategies that can be used to test software. At one extreme, you can wait until the system is fully constructed and then conduct tests on the overall sys- tem in hopes of finding errors. This approach, although appealing, simply does not work. It will result in buggy software that disappoints all stakeholders. At the other extreme, you could conduct tests on a daily basis, whenever any part of the system is constructed. This approach, although less appealing to many, can be very effec- tive. Unfortunately, some software developers hesitate to use it. What to do?

A testing strategy that is chosen by most software teams falls between the two extremes. It takes an incremental view of testing, beginning with the testing of indi- vidual program units, moving to tests designed to facilitate the integration of the units, and culminating with tests that exercise the constructed system. Each of these classes of tests is described in the sections that follow.

17.3.1 Unit Testing

Unit testing focuses verification effort on the smallest unit of software design—the software component or module. Using the component-level design description as a

456 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

uote:

“Testing only to end-user requirements is like inspecting a building based on the work done by the interior designer at the expense of the foundations, girders, and plumbing.”

Boris Beizer

2 Throughout this book, I use the terms conventional software or traditional software to refer to com- mon hierarchical or call-and-return software architectures that are frequently encountered in a variety of application domains. Traditional software architectures are not object-oriented and do not encompass WebApps.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 456

guide, important control paths are tested to uncover errors within the boundary of the module. The relative complexity of tests and the errors those tests uncover is lim- ited by the constrained scope established for unit testing. The unit test focuses on the internal processing logic and data structures within the boundaries of a component. This type of testing can be conducted in parallel for multiple components.

Unit-test considerations. Unit tests are illustrated schematically in Figure 17.3. The module interface is tested to ensure that information properly flows into and out of the program unit under test. Local data structures are examined to ensure that data stored temporarily maintains its integrity during all steps in an algorithm’s execution. All independent paths through the control structure are exercised to ensure that all statements in a module have been executed at least once. Boundary conditions are tested to ensure that the module operates properly at boundaries established to limit or restrict processing. And finally, all error-handling paths are tested.

Data flow across a component interface is tested before any other testing is initi- ated. If data do not enter and exit properly, all other tests are moot. In addition, local data structures should be exercised and the local impact on global data should be as- certained (if possible) during unit testing.

Selective testing of execution paths is an essential task during the unit test. Test cases should be designed to uncover errors due to erroneous computations, incor- rect comparisons, or improper control flow.

Boundary testing is one of the most important unit testing tasks. Software often fails at its boundaries. That is, errors often occur when the nth element of an n-dimensional array is processed, when the i th repetition of a loop with i passes is invoked, when the maximum or minimum allowable value is encountered. Test cases that exercise data structure, control flow, and data values just below, at, and just above maxima and minima are very likely to uncover errors.

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 457

Test cases

Module Interface Local data structures Boundary conditions Independent paths Error-handling paths

FIGURE 17.3 Unit test

It’s not a bad idea to design unit test cases before you develop code for a component. It helps ensure that you’ll develop code that will pass the tests.

What errors are commonly

found during unit testing?

?

pre75977_ch17.qxd 11/27/08 6:09 PM Page 457

A good design anticipates error conditions and establishes error-handling paths to reroute or cleanly terminate processing when an error does occur. Yourdon [You75] calls this approach antibugging. Unfortunately, there is a tendency to incor- porate error handling into software and then never test it. A true story may serve to illustrate:

A computer-aided design system was developed under contract. In one transaction pro-

cessing module, a practical joker placed the following error handling message after a

series of conditional tests that invoked various control flow branches: ERROR! THERE IS

NO WAY YOU CAN GET HERE. This “error message” was uncovered by a customer during

user training!

Among the potential errors that should be tested when error handling is evalu- ated are: (1) error description is unintelligible, (2) error noted does not correspond to error encountered, (3) error condition causes system intervention prior to error han- dling, (4) exception-condition processing is incorrect, or (5) error description does not provide enough information to assist in the location of the cause of the error.

Unit-test procedures. Unit testing is normally considered as an adjunct to the coding step. The design of unit tests can occur before coding begins or after source code has been generated. A review of design information provides guidance for establishing test cases that are likely to uncover errors in each of the categories dis- cussed earlier. Each test case should be coupled with a set of expected results.

Because a component is not a stand-alone program, driver and/or stub software must often be developed for each unit test. The unit test environment is illustrated in Figure 17.4. In most applications a driver is nothing more than a “main program” that accepts test case data, passes such data to the component (to be tested), and prints

458 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

WebRef Useful information on a wide variety of articles and resources for “agile testing” can be found at testing .com/agile/.

Be sure that you design tests to execute every error-handling path. If you don’t, the path may fail when it is invoked, exacer- bating an already dicey situation.

Test cases

Interface Local data structures Boundary conditions Independent paths Error-handling paths

Module to be tested

Stub Stub

Driver

RESULTS

FIGURE 17.4 Unit-test environment

pre75977_ch17.qxd 11/27/08 6:09 PM Page 458

relevant results. Stubs serve to replace modules that are subordinate (invoked by) the component to be tested. A stub or “dummy subprogram” uses the subordinate mod- ule’s interface, may do minimal data manipulation, prints verification of entry, and returns control to the module undergoing testing.

Drivers and stubs represent testing “overhead.” That is, both are software that must be written (formal design is not commonly applied) but that is not delivered with the final software product. If drivers and stubs are kept simple, actual overhead is rel- atively low. Unfortunately, many components cannot be adequately unit tested with “simple” overhead software. In such cases, complete testing can be postponed until the integration test step (where drivers or stubs are also used).

Unit testing is simplified when a component with high cohesion is designed. When only one function is addressed by a component, the number of test cases is reduced and errors can be more easily predicted and uncovered.

17.3.2 Integration Testing

A neophyte in the software world might ask a seemingly legitimate question once all modules have been unit tested: “If they all work individually, why do you doubt that they’ll work when we put them together?” The problem, of course, is “putting them together”—interfacing. Data can be lost across an interface; one component can have an inadvertent, adverse effect on another; subfunctions, when combined, may not produce the desired major function; individually acceptable imprecision may be magnified to unacceptable levels; global data structures can present problems. Sadly, the list goes on and on.

Integration testing is a systematic technique for constructing the software archi- tecture while at the same time conducting tests to uncover errors associated with interfacing. The objective is to take unit-tested components and build a program structure that has been dictated by design.

There is often a tendency to attempt nonincremental integration; that is, to con- struct the program using a “big bang” approach. All components are combined in advance. The entire program is tested as a whole. And chaos usually results! A set of errors is encountered. Correction is difficult because isolation of causes is com- plicated by the vast expanse of the entire program. Once these errors are corrected, new ones appear and the process continues in a seemingly endless loop.

Incremental integration is the antithesis of the big bang approach. The program is constructed and tested in small increments, where errors are easier to isolate and correct; interfaces are more likely to be tested completely; and a systematic test approach may be applied. In the paragraphs that follow, a number of different incre- mental integration strategies are discussed.

Top-down integration. Top-down integration testing is an incremental approach to construction of the software architecture. Modules are integrated by moving downward through the control hierarchy, beginning with the main control module

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 459

There are some situa- tions in which you will not have the resources to do comprehensive unit testing. Select critical or complex modules and unit test only those.

Taking the “big bang” approach to integration is a lazy strategy that is doomed to failure. Integrate incremen- tally, testing as you go.

When you develop a project schedule, you’ll have to consider the manner in which inte- gration will occur so that components will be available when needed.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 459

(main program). Modules subordinate (and ultimately subordinate) to the main con- trol module are incorporated into the structure in either a depth-first or breadth-first manner.

Referring to Figure 17.5, depth-first integration integrates all components on a major control path of the program structure. Selection of a major path is somewhat arbitrary and depends on application-specific characteristics. For example, selecting the left-hand path, components M1, M2 , M5 would be integrated first. Next, M8 or (if necessary for proper functioning of M2) M6 would be integrated. Then, the central and right-hand control paths are built. Breadth-first integration incorporates all com- ponents directly subordinate at each level, moving across the structure horizontally. From the figure, components M2, M3, and M4 would be integrated first. The next con- trol level, M5, M6, and so on, follows. The integration process is performed in a series of five steps:

1. The main control module is used as a test driver and stubs are substituted for all components directly subordinate to the main control module.

2. Depending on the integration approach selected (i.e., depth or breadth first), subordinate stubs are replaced one at a time with actual components.

3. Tests are conducted as each component is integrated.

4. On completion of each set of tests, another stub is replaced with the real component.

5. Regression testing (discussed later in this section) may be conducted to ensure that new errors have not been introduced.

The process continues from step 2 until the entire program structure is built.

460 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

What are the steps for

top-down integration?

?

M1

M3M2

M7M6M5

M8

M4

FIGURE 17.5 Top-down integration

pre75977_ch17.qxd 11/27/08 6:09 PM Page 460

The top-down integration strategy verifies major control or decision points early in the test process. In a “well-factored” program structure, decision making occurs at upper levels in the hierarchy and is therefore encountered first. If major control problems do exist, early recognition is essential. If depth-first integration is selected, a complete function of the software may be implemented and demonstrated. Early demonstration of functional capability is a confidence builder for all stakeholders.

Top-down strategy sounds relatively uncomplicated, but in practice, logistical problems can arise. The most common of these problems occurs when processing at low levels in the hierarchy is required to adequately test upper levels. Stubs replace low-level modules at the beginning of top-down testing; therefore, no sig- nificant data can flow upward in the program structure. As a tester, you are left with three choices: (1) delay many tests until stubs are replaced with actual modules, (2) develop stubs that perform limited functions that simulate the actual module, or (3) integrate the software from the bottom of the hierarchy upward.

The first approach (delay tests until stubs are replaced by actual modules) can cause you to lose some control over correspondence between specific tests and incorporation of specific modules. This can lead to difficulty in determining the cause of errors and tends to violate the highly constrained nature of the top-down approach. The second approach is workable but can lead to significant overhead, as stubs become more and more complex. The third approach, called bottom-up inte- gration is discussed in the paragraphs that follow.

Bottom-up integration. Bottom-up integration testing, as its name implies, begins construction and testing with atomic modules (i.e., components at the lowest levels in the program structure). Because components are integrated from the bottom up, the functionality provided by components subordinate to a given level is always available and the need for stubs is eliminated. A bottom-up integration strategy may be implemented with the following steps:

1. Low-level components are combined into clusters (sometimes called builds) that perform a specific software subfunction.

2. A driver (a control program for testing) is written to coordinate test case input and output.

3. The cluster is tested.

4. Drivers are removed and clusters are combined moving upward in the program structure.

Integration follows the pattern illustrated in Figure 17.6. Components are com- bined to form clusters 1, 2, and 3. Each of the clusters is tested using a driver (shown as a dashed block). Components in clusters 1 and 2 are subordinate to Ma. Drivers D1 and D2 are removed and the clusters are interfaced directly to Ma. Similarly, driver D3 for cluster 3 is removed prior to integration with module Mb. Both Ma and Mb will ultimately be integrated with component Mc, and so forth.

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 461

What problems

may be encountered when top-down integration is chosen?

?

What are the steps for

bottom-up integration?

?

Bottom-up integration eliminates the need for complex stubs.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 461

As integration moves upward, the need for separate test drivers lessens. In fact, if the top two levels of program structure are integrated top down, the number of drivers can be reduced substantially and integration of clusters is greatly simplified.

Regression testing. Each time a new module is added as part of integration test- ing, the software changes. New data flow paths are established, new I/O may occur, and new control logic is invoked. These changes may cause problems with functions that previously worked flawlessly. In the context of an integration test strategy, regression testing is the reexecution of some subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects.

In a broader context, successful tests (of any kind) result in the discovery of errors, and errors must be corrected. Whenever software is corrected, some aspect of the software configuration (the program, its documentation, or the data that support it) is changed. Regression testing helps to ensure that changes (due to testing or for other reasons) do not introduce unintended behavior or additional errors.

Regression testing may be conducted manually, by reexecuting a subset of all test cases or using automated capture/playback tools. Capture/playback tools enable the software engineer to capture test cases and results for subsequent playback and comparison. The regression test suite (the subset of tests to be executed) contains three different classes of test cases:

• A representative sample of tests that will exercise all software functions.

• Additional tests that focus on software functions that are likely to be affected by the change.

• Tests that focus on the software components that have been changed.

462 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Mc

Ma

D2 D3D1

Mb

Cluster 1

Cluster 3

Cluster 2

FIGURE 17.6 Bottom-up integration

Regression testing is an important strategy for reducing “side effects.” Run regres- sion tests every time a major change is made to the software (including the integration of new components).

pre75977_ch17.qxd 11/27/08 6:09 PM Page 462

As integration testing proceeds, the number of regression tests can grow quite large. Therefore, the regression test suite should be designed to include only those tests that address one or more classes of errors in each of the major program func- tions. It is impractical and inefficient to reexecute every test for every program func- tion once a change has occurred.

Smoke testing. Smoke testing is an integration testing approach that is com- monly used when product software is developed. It is designed as a pacing mecha- nism for time-critical projects, allowing the software team to assess the project on a frequent basis. In essence, the smoke-testing approach encompasses the follow- ing activities:

1. Software components that have been translated into code are integrated into a build. A build includes all data files, libraries, reusable modules, and engi- neered components that are required to implement one or more product functions.

2. A series of tests is designed to expose errors that will keep the build from properly performing its function. The intent should be to uncover “show- stopper” errors that have the highest likelihood of throwing the software project behind schedule.

3. The build is integrated with other builds, and the entire product (in its current form) is smoke tested daily. The integration approach may be top down or bottom up.

The daily frequency of testing the entire product may surprise some readers. How- ever, frequent tests give both managers and practitioners a realistic assessment of integration testing progress. McConnell [McC96] describes the smoke test in the following manner:

The smoke test should exercise the entire system from end to end. It does not have to be

exhaustive, but it should be capable of exposing major problems. The smoke test should

be thorough enough that if the build passes, you can assume that it is stable enough to

be tested more thoroughly.

Smoke testing provides a number of benefits when it is applied on complex, time- critical software projects:

• Integration risk is minimized. Because smoke tests are conducted daily, incompatibilities and other show-stopper errors are uncovered early, thereby reducing the likelihood of serious schedule impact when errors are uncovered.

• The quality of the end product is improved. Because the approach is construc- tion (integration) oriented, smoke testing is likely to uncover functional errors as well as architectural and component-level design errors. If these errors are corrected early, better product quality will result.

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 463

Smoke testing might be characterized as a rolling integration strategy. The software is rebuilt (with new components added) and smoke tested every day.

uote:

“Treat the daily build as the heartbeat of the project. If there’s no heartbeat, the project is dead.”

Jim McCarthy

What benefits can

be derived from smoke testing?

?

pre75977_ch17.qxd 11/27/08 6:09 PM Page 463

• Error diagnosis and correction are simplified. Like all integration testing approaches, errors uncovered during smoke testing are likely to be associ- ated with “new software increments”—that is, the software that has just been added to the build(s) is a probable cause of a newly discovered error.

• Progress is easier to assess. With each passing day, more of the software has been integrated and more has been demonstrated to work. This improves team morale and gives managers a good indication that progress is being made.

Strategic options. There has been much discussion (e.g., [Bei84]) about the rela- tive advantages and disadvantages of top-down versus bottom-up integration test- ing. In general, the advantages of one strategy tend to result in disadvantages for the other strategy. The major disadvantage of the top-down approach is the need for stubs and the attendant testing difficulties that can be associated with them. Prob- lems associated with stubs may be offset by the advantage of testing major control functions early. The major disadvantage of bottom-up integration is that “the pro- gram as an entity does not exist until the last module is added” [Mye79]. This draw- back is tempered by easier test case design and a lack of stubs.

Selection of an integration strategy depends upon software characteristics and, sometimes, project schedule. In general, a combined approach (sometimes called sandwich testing) that uses top-down tests for upper levels of the program structure, coupled with bottom-up tests for subordinate levels may be the best compromise.

As integration testing is conducted, the tester should identify critical modules. A critical module has one or more of the following characteristics: (1) addresses several software requirements, (2) has a high level of control (resides relatively high in the program structure), (3) is complex or error prone, or (4) has definite performance requirements. Critical modules should be tested as early as is possible. In addition, regression tests should focus on critical module function.

Integration test work products. An overall plan for integration of the software and a description of specific tests is documented in a Test Specification. This work prod- uct incorporates a test plan and a test procedure and becomes part of the software configuration. Testing is divided into phases and builds that address specific func- tional and behavioral characteristics of the software. For example, integration testing for the SafeHome security system might be divided into the following test phases:

• User interaction (command input and output, display representation, error processing and representation)

• Sensor processing (acquisition of sensor output, determination of sensor conditions, actions required as a consequence of conditions)

• Communications functions (ability to communicate with central monitoring station)

• Alarm processing (tests of software actions that occur when an alarm is encountered)

464 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

WebRef Pointers to commentary on testing strategies can be found at www.qalinks.com.

What is a “critical

module” and why should we identify it?

?

pre75977_ch17.qxd 11/27/08 6:09 PM Page 464

Each of these integration test phases delineates a broad functional category within the software and generally can be related to a specific domain within the soft- ware architecture. Therefore, program builds (groups of modules) are created to cor- respond to each phase. The following criteria and corresponding tests are applied for all test phases:

Interface integrity. Internal and external interfaces are tested as each module (or cluster) is incorporated into the structure.

Functional validity. Tests designed to uncover functional errors are conducted.

Information content. Tests designed to uncover errors associated with local or global data structures are conducted.

Performance. Tests designed to verify performance bounds established during software design are conducted.

A schedule for integration, the development of overhead software, and related topics are also discussed as part of the test plan. Start and end dates for each phase are established and “availability windows” for unit-tested modules are defined. A brief description of overhead software (stubs and drivers) concentrates on charac- teristics that might require special effort. Finally, test environment and resources are described. Unusual hardware configurations, exotic simulators, and special test tools or techniques are a few of many topics that may also be discussed.

The detailed testing procedure that is required to accomplish the test plan is described next. The order of integration and corresponding tests at each integration step are described. A listing of all test cases (annotated for subsequent reference) and expected results are also included.

A history of actual test results, problems, or peculiarities is recorded in a Test Report that can be appended to the Test Specification, if desired. Information con- tained in this section can be vital during software maintenance. Appropriate refer- ences and appendixes are also presented.

Like all other elements of a software configuration, the test specification format may be tailored to the local needs of a software engineering organization. It is impor- tant to note, however, that an integration strategy (contained in a test plan) and test- ing details (described in a test procedure) are essential ingredients and must appear.

1 7 . 4 T E S T S T R AT E G I E S F O R O B J E C T - O R I E N T E D S O F T WA R E 3

The objective of testing, stated simply, is to find the greatest possible number of errors with a manageable amount of effort applied over a realistic time span. Although this fundamental objective remains unchanged for object-oriented software, the nature of object-oriented software changes both testing strategy and testing tactics (Chapter 19).

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 465

3 Basic object-oriented concepts are presented in Appendix 2.

What criteria should be

used to design integration tests?

?

pre75977_ch17.qxd 11/27/08 6:09 PM Page 465

17.4.1 Unit Testing in the OO Context

When object-oriented software is considered, the concept of the unit changes. Encapsulation drives the definition of classes and objects. This means that each class and each instance of a class packages attributes (data) and the operations that manipulate these data. An encapsulated class is usually the focus of unit testing. How- ever, operations (methods) within the class are the smallest testable units. Because a class can contain a number of different operations, and a particular operation may exist as part of a number of different classes, the tactics applied to unit testing must change.

You can no longer test a single operation in isolation (the conventional view of unit testing) but rather as part of a class. To illustrate, consider a class hierarchy in which an operation X is defined for the superclass and is inherited by a number of subclasses. Each subclass uses operation X, but it is applied within the context of the private attributes and operations that have been defined for the subclass. Because the context in which operation X is used varies in subtle ways, it is necessary to test operation X in the context of each of the subclasses. This means that testing opera- tion X in a stand-alone fashion (the conventional unit-testing approach) is usually ineffective in the object-oriented context.

Class testing for OO software is the equivalent of unit testing for conventional software. Unlike unit testing of conventional software, which tends to focus on the algorithmic detail of a module and the data that flow across the module interface, class testing for OO software is driven by the operations encapsulated by the class and the state behavior of the class.

17.4.2 Integration Testing in the OO Context

Because object-oriented software does not have an obvious hierarchical control structure, traditional top-down and bottom-up integration strategies (Section 17.3.2) have little meaning. In addition, integrating operations one at a time into a class (the conventional incremental integration approach) is often impossible because of the “direct and indirect interactions of the components that make up the class” [Ber93].

There are two different strategies for integration testing of OO systems [Bin94b]. The first, thread-based testing, integrates the set of classes required to respond to one input or event for the system. Each thread is integrated and tested individually. Regression testing is applied to ensure that no side effects occur. The second inte- gration approach, use-based testing, begins the construction of the system by testing those classes (called independent classes) that use very few (if any) server classes. After the independent classes are tested, the next layer of classes, called dependent classes, that use the independent classes are tested. This sequence of testing layers of dependent classes continues until the entire system is constructed.

The use of drivers and stubs also changes when integration testing of OO systems is conducted. Drivers can be used to test operations at the lowest level and for the testing of whole groups of classes. A driver can also be used to replace the user inter- face so that tests of system functionality can be conducted prior to implementation

466 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Class testing for OO software is analogous to module testing for conventional software. It is not advisable to test operations in isolation.

An important strategy for integration testing of OO software is thread-based testing. Threads are sets of classes that respond to an input or event. Use- based tests focus on classes that do not collaborate heavily with other classes.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 466

of the interface. Stubs can be used in situations in which collaboration between classes is required but one or more of the collaborating classes has not yet been fully implemented.

Cluster testing is one step in the integration testing of OO software. Here, a cluster of collaborating classes (determined by examining the CRC and object-relationship model) is exercised by designing test cases that attempt to uncover errors in the collaborations.

1 7 . 5 T E S T S T R AT E G I E S F O R W E B A P P S The strategy for WebApp testing adopts the basic principles for all software testing and applies a strategy and tactics that are used for object-oriented systems. The following steps summarize the approach:

1. The content model for the WebApp is reviewed to uncover errors.

2. The interface model is reviewed to ensure that all use cases can be accommodated.

3. The design model for the WebApp is reviewed to uncover navigation errors.

4. The user interface is tested to uncover errors in presentation and/or naviga- tion mechanics.

5. Each functional component is unit tested.

6. Navigation throughout the architecture is tested.

7. The WebApp is implemented in a variety of different environmental configu- rations and is tested for compatibility with each configuration.

8. Security tests are conducted in an attempt to exploit vulnerabilities in the WebApp or within its environment.

9. Performance tests are conducted.

10. The WebApp is tested by a controlled and monitored population of end users. The results of their interaction with the system are evaluated for content and navigation errors, usability concerns, compatibility concerns, and WebApp reliability and performance.

Because many WebApps evolve continuously, the testing process is an ongoing activity, conducted by support staff who use regression tests derived from the tests developed when the WebApp was first engineered. Methods for WebApp testing are considered in Chapter 20.

1 7 . 6 VA L I D AT I O N T E S T I N G Validation testing begins at the culmination of integration testing, when individual components have been exercised, the software is completely assembled as a pack- age, and interfacing errors have been uncovered and corrected. At the validation or system level, the distinction between conventional software, object-oriented

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 467

The overall strategy for WebApp testing can be summarized in the 10 steps noted here.

WebRef Excellent articles on WebApp testing can be found at www .stickyminds.com/ testing.asp.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 467

software, and WebApps disappears. Testing focuses on user-visible actions and user-recognizable output from the system.

Validation can be defined in many ways, but a simple (albeit harsh) definition is that validation succeeds when software functions in a manner that can be reason- ably expected by the customer. At this point a battle-hardened software developer might protest: “Who or what is the arbiter of reasonable expectations?” If a Software Requirements Specification has been developed, it describes all user-visible attributes of the software and contains a Validation Criteria section that forms the basis for a validation-testing approach.

17.6.1 Validation-Test Criteria

Software validation is achieved through a series of tests that demonstrate conform- ity with requirements. A test plan outlines the classes of tests to be conducted, and a test procedure defines specific test cases that are designed to ensure that all func- tional requirements are satisfied, all behavioral characteristics are achieved, all con- tent is accurate and properly presented, all performance requirements are attained, documentation is correct, and usability and other requirements are met (e.g., trans- portability, compatibility, error recovery, maintainability).

After each validation test case has been conducted, one of two possible condi- tions exists: (1) The function or performance characteristic conforms to specification and is accepted or (2) a deviation from specification is uncovered and a deficiency list is created. Deviations or errors discovered at this stage in a project can rarely be corrected prior to scheduled delivery. It is often necessary to negotiate with the cus- tomer to establish a method for resolving deficiencies.

17.6.2 Configuration Review

An important element of the validation process is a configuration review. The intent of the review is to ensure that all elements of the software configuration have been properly developed, are cataloged, and have the necessary detail to bolster the sup- port activities. The configuration review, sometimes called an audit, is discussed in more detail in Chapter 22.

17.6.3 Alpha and Beta Testing

It is virtually impossible for a software developer to foresee how the customer will really use a program. Instructions for use may be misinterpreted; strange combina- tions of data may be regularly used; output that seemed clear to the tester may be unintelligible to a user in the field.

When custom software is built for one customer, a series of acceptance tests are conducted to enable the customer to validate all requirements. Conducted by the end user rather than software engineers, an acceptance test can range from an informal “test drive” to a planned and systematically executed series of tests. In fact, accept- ance testing can be conducted over a period of weeks or months, thereby uncover- ing cumulative errors that might degrade the system over time.

468 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Like all other testing steps, validation tries to uncover errors, but the focus is at the requirements level— on things that will be immediately apparent to the end user.

uote:

“Given enough eyeballs, all bugs are shallow (e.g., given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone).”

E. Raymond

pre75977_ch17.qxd 11/27/08 6:09 PM Page 468

If software is developed as a product to be used by many customers, it is imprac- tical to perform formal acceptance tests with each one. Most software product builders use a process called alpha and beta testing to uncover errors that only the end user seems able to find.

The alpha test is conducted at the developer’s site by a representative group of end users. The software is used in a natural setting with the developer “looking over the shoulder” of the users and recording errors and usage problems. Alpha tests are con- ducted in a controlled environment.

The beta test is conducted at one or more end-user sites. Unlike alpha testing, the developer generally is not present. Therefore, the beta test is a “live” application of the software in an environment that cannot be controlled by the developer. The cus- tomer records all problems (real or imagined) that are encountered during beta test- ing and reports these to the developer at regular intervals. As a result of problems reported during beta tests, you make modifications and then prepare for release of the software product to the entire customer base.

A variation on beta testing, called customer acceptance testing, is sometimes per- formed when custom software is delivered to a customer under contract. The cus- tomer performs a series of specific tests in an attempt to uncover errors before accepting the software from the developer. In some cases (e.g., a major corporate or governmental system) acceptance testing can be very formal and encompass many days or even weeks of testing.

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 469

What is the difference

between an alpha test and a beta test?

?

The scene: Doug Miller’s office, as component-level design continues and construction of certain components continues.

The players: Doug Miller, software engineering manager, Vinod, Jamie, Ed, and Shakira—members of the SafeHome software engineering team.

The conversation:

Doug: The first increment will be ready for validation in what . . . about three weeks?

Vinod: That’s about right. Integration is going well. We’re smoke testing daily, finding some bugs but nothing we can’t handle. So far, so good.

Doug: Talk to me about validation.

Shakira: Well, we’ll use all of the use cases as the basis for our test design. I haven’t started yet, but I’ll be developing tests for all of the use cases that I’ve been responsible for.

Ed: Same here.

Jamie: Me too, but we’ve got to get our act together for acceptance test and also for alpha and beta testing, no?

Doug: Yes. In fact I’ve been thinking; we could bring in an outside contractor to help us with validation. I have the money in the budget . . . and it’d give us a new point of view.

Vinod: I think we’ve got it under control.

Doug: I’m sure you do, but an ITG gives us an independent look at the software.

Jamie: We’re tight on time here, Doug. I for one don’t have the time to babysit anybody you bring in to do the job.

Doug: I know, I know. But if an ITG works from requirements and use cases, not too much babysitting will be required.

Vinod: I still think we’ve got it under control.

Doug: I hear you, Vinod, but I going to overrule on this one. Let’s plan to meet with the ITG rep later this week. Get ‘em started and see what they come up with.

Vinod: Okay, maybe it’ll lighten the load a bit.

SAFEHOME

Preparing for Validation

pre75977_ch17.qxd 11/27/08 6:09 PM Page 469

1 7 . 7 S Y S T E M T E S T I N G At the beginning of this book, I stressed the fact that software is only one element of a larger computer-based system. Ultimately, software is incorporated with other sys- tem elements (e.g., hardware, people, information), and a series of system integra- tion and validation tests are conducted. These tests fall outside the scope of the software process and are not conducted solely by software engineers. However, steps taken during software design and testing can greatly improve the probability of successful software integration in the larger system.

A classic system-testing problem is “finger pointing.” This occurs when an error is uncovered, and the developers of different system elements blame each other for the problem. Rather than indulging in such nonsense, you should anticipate poten- tial interfacing problems and (1) design error-handling paths that test all information coming from other elements of the system, (2) conduct a series of tests that simulate bad data or other potential errors at the software interface, (3) record the results of tests to use as “evidence” if finger pointing does occur, and (4) participate in plan- ning and design of system tests to ensure that software is adequately tested.

System testing is actually a series of different tests whose primary purpose is to fully exercise the computer-based system. Although each test has a different pur- pose, all work to verify that system elements have been properly integrated and per- form allocated functions. In the sections that follow, I discuss the types of system tests that are worthwhile for software-based systems.

17.7.1 Recovery Testing

Many computer-based systems must recover from faults and resume processing with little or no downtime. In some cases, a system must be fault tolerant; that is, processing faults must not cause overall system function to cease. In other cases, a system failure must be corrected within a specified period of time or severe eco- nomic damage will occur.

Recovery testing is a system test that forces the software to fail in a variety of ways and verifies that recovery is properly performed. If recovery is automatic (performed by the system itself), reinitialization, checkpointing mechanisms, data recovery, and restart are evaluated for correctness. If recovery requires human intervention, the mean-time-to-repair (MTTR) is evaluated to determine whether it is within accept- able limits.

17.7.2 Security Testing

Any computer-based system that manages sensitive information or causes actions that can improperly harm (or benefit) individuals is a target for improper or illegal penetration. Penetration spans a broad range of activities: hackers who attempt to penetrate systems for sport, disgruntled employees who attempt to penetrate for revenge, dishonest individuals who attempt to penetrate for illicit personal gain.

470 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

uote:

“Like death and taxes, testing is both unpleasant and inevitable.”

Ed Yourdon

pre75977_ch17.qxd 11/27/08 6:09 PM Page 470

Security testing attempts to verify that protection mechanisms built into a system will, in fact, protect it from improper penetration. To quote Beizer [Bei84]: “The sys- tem’s security must, of course, be tested for invulnerability from frontal attack—but must also be tested for invulnerability from flank or rear attack.”

During security testing, the tester plays the role(s) of the individual who desires to penetrate the system. Anything goes! The tester may attempt to acquire passwords through external clerical means; may attack the system with custom software designed to break down any defenses that have been constructed; may overwhelm the system, thereby denying service to others; may purposely cause system errors, hoping to penetrate during recovery; may browse through insecure data, hoping to find the key to system entry.

Given enough time and resources, good security testing will ultimately penetrate a system. The role of the system designer is to make penetration cost more than the value of the information that will be obtained.

17.7.3 Stress Testing

Earlier software testing steps resulted in thorough evaluation of normal program functions and performance. Stress tests are designed to confront programs with abnormal situations. In essence, the tester who performs stress testing asks: “How high can we crank this up before it fails?”

Stress testing executes a system in a manner that demands resources in abnor- mal quantity, frequency, or volume. For example, (1) special tests may be designed that generate ten interrupts per second, when one or two is the average rate, (2) input data rates may be increased by an order of magnitude to determine how input functions will respond, (3) test cases that require maximum memory or other resources are executed, (4) test cases that may cause thrashing in a virtual oper- ating system are designed, (5) test cases that may cause excessive hunting for disk-resident data are created. Essentially, the tester attempts to break the program.

A variation of stress testing is a technique called sensitivity testing. In some situa- tions (the most common occur in mathematical algorithms), a very small range of data contained within the bounds of valid data for a program may cause extreme and even erroneous processing or profound performance degradation. Sensitivity testing attempts to uncover data combinations within valid input classes that may cause instability or improper processing.

17.7.4 Performance Testing

For real-time and embedded systems, software that provides required function but does not conform to performance requirements is unacceptable. Performance test- ing is designed to test the run-time performance of software within the context of an integrated system. Performance testing occurs throughout all steps in the testing process. Even at the unit level, the performance of an individual module may be

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 471

uote:

“If you’re trying to find true system bugs and you haven’t subjected your software to a real stress test, then it’s high time you started.”

Boris Beizer

pre75977_ch17.qxd 11/27/08 6:09 PM Page 471

assessed as tests are conducted. However, it is not until all system elements are fully integrated that the true performance of a system can be ascertained.

Performance tests are often coupled with stress testing and usually require both hardware and software instrumentation. That is, it is often necessary to measure resource utilization (e.g., processor cycles) in an exacting fashion. External instru- mentation can monitor execution intervals, log events (e.g., interrupts) as they oc- cur, and sample machine states on a regular basis. By instrumenting a system, the tester can uncover situations that lead to degradation and possible system failure.

17.7.5 Deployment Testing

In many cases, software must execute on a variety of platforms and under more than one operating system environment. Deployment testing, sometimes called configuration testing, exercises the software in each environment in which it is to operate. In addition, deployment testing examines all installation procedures and specialized installation software (e.g., “installers”) that will be used by customers, and all documentation that will be used to introduce the software to end users.

As an example, consider the Internet-accessible version of SafeHome software that would allow a customer to monitor the security system from remote locations. The SafeHome WebApp must be tested using all Web browsers that are likely to be encountered. A more thorough deployment test might encompass combinations of Web browsers with various operating systems (e.g., Linux, Mac OS, Windows). Because security is a major issue, a complete set of security tests would be integrated with the deployment test.

472 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Test Planning and Management Objective: These tools assist a software team in planning the testing strategy that is chosen

and managing the testing process as it is conducted.

Mechanics: Tools in this category address test planning, test storage, management and control, requirements traceability, integration, error tracking, and report generation. Project managers use them to supplement project scheduling tools. Testers use these tools to plan testing activities and to control the flow of information as the testing process proceeds.

Representative Tools:4

QaTraq Test Case Management Tool, developed by Traq Software (www.testmanagement.com), “encourages a structured approach to test management.”

QADirector, developed by Compuware Corp. (www.compuware.com/qacenter), provides a single point of control for managing all phases of the testing process.

TestWorks, developed by Software Research, Inc. (www.soft.com/Products/index.html), contains a fully integrated suite of testing tools including tools for test management and reporting.

OpensourceTesting.org (www.opensourcetesting.org/testmgt.php) lists a variety of open-source test management and planning tools.

NI TestStand, developed by National Instruments Corp. (www.ni.com), allows you to “develop, manage, and execute test sequences written in any programming language.”

SOFTWARE TOOLS

4 Tools noted here do not represent an endorsement, but rather a sampling of tools in this category. In most cases, tool names are trademarked by their respective developers.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 472

1 7 . 8 T H E A R T O F D E B U G G I N G Software testing is a process that can be systematically planned and specified. Test- case design can be conducted, a strategy can be defined, and results can be evalu- ated against prescribed expectations.

Debugging occurs as a consequence of successful testing. That is, when a test case uncovers an error, debugging is the process that results in the removal of the error. Although debugging can and should be an orderly process, it is still very much an art. As a software engineer, you are often confronted with a “symptomatic” indi- cation of a software problem as you evaluate the results of a test. That is, the exter- nal manifestation of the error and its internal cause may have no obvious relationship to one another. The poorly understood mental process that connects a symptom to a cause is debugging.

17.8.1 The Debugging Process

Debugging is not testing but often occurs as a consequence of testing.5 Referring to Figure 17.7, the debugging process begins with the execution of a test case. Results are assessed and a lack of correspondence between expected and actual perform- ance is encountered. In many cases, the noncorresponding data are a symptom of an underlying cause as yet hidden. The debugging process attempts to match symp- tom with cause, thereby leading to error correction.

The debugging process will usually have one of two outcomes: (1) the cause will be found and corrected or (2) the cause will not be found. In the latter case, the per- son performing debugging may suspect a cause, design a test case to help validate that suspicion, and work toward error correction in an iterative fashion.

Why is debugging so difficult? In all likelihood, human psychology (see Sec- tion 17.8.2) has more to do with an answer than software technology. However, a few characteristics of bugs provide some clues:

1. The symptom and the cause may be geographically remote. That is, the symptom may appear in one part of a program, while the cause may actually be located at a site that is far removed. Highly coupled components (Chapter 8) exacerbate this situation.

2. The symptom may disappear (temporarily) when another error is corrected.

3. The symptom may actually be caused by nonerrors (e.g., round-off inaccuracies).

4. The symptom may be caused by human error that is not easily traced.

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 473

uote:

“We found to our surprise that it wasn’t as easy to get programs right as we had thought. I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.”

Maurice Wilkes, discovers debugging, 1949

5 In making the statement, we take the broadest possible view of testing. Not only does the devel- oper test software prior to release, but the customer/user tests software every time it is used!

Be certain to avoid a third outcome: The cause is found, but the “correction” does not solve the problem or introduces still another error.

Why is debugging so

difficult? ?

pre75977_ch17.qxd 11/27/08 6:09 PM Page 473

5. The symptom may be a result of timing problems, rather than processing problems.

6. It may be difficult to accurately reproduce input conditions (e.g., a real-time application in which input ordering is indeterminate).

7. The symptom may be intermittent. This is particularly common in embedded systems that couple hardware and software inextricably.

8. The symptom may be due to causes that are distributed across a number of tasks running on different processors.

During debugging, you’ll encounter errors that range from mildly annoying (e.g., an incorrect output format) to catastrophic (e.g., the system fails, causing serious economic or physical damage). As the consequences of an error increase, the amount of pressure to find the cause also increases. Often, pressure forces some software developers to fix one error and at the same time introduce two more.

17.8.2 Psychological Considerations

Unfortunately, there appears to be some evidence that debugging prowess is an innate human trait. Some people are good at it and others aren’t. Although experimental evidence on debugging is open to many interpretations, large variances in debugging

474 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Regression tests

Corrections

Identified causes

Additional tests

Suspected causes

Results

Debugging

Test Cases

FIGURE 17.7 The debugging process

“Everyone knows that

debugging is twice as hard as writing a pro- gram in the first place. So if you are as clever as you can be when you write it, how will you ever debug it?”

Brian Kernighan

?

pre75977_ch17.qxd 11/27/08 6:09 PM Page 474

ability have been reported for programmers with the same education and experience. Commenting on the human aspects of debugging, Shneiderman [Shn80] states:

Debugging is one of the more frustrating parts of programming. It has elements of prob-

lem solving or brain teasers, coupled with the annoying recognition that you have made

a mistake. Heightened anxiety and the unwillingness to accept the possibility of errors

increases the task difficulty. Fortunately, there is a great sigh of relief and a lessening of

tension when the bug is ultimately . . . corrected.

Although it may be difficult to “learn” debugging, a number of approaches to the problem can be proposed. I examine them in Section 17.8.3.

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 475

Set a time limit, say two hours, on the amount of time you spend trying to debug a problem on your own. After that, get help!

Debugging

The scene: Ed’s cubical as code and unit testing is conducted.

The players: Ed and Shakira—members of the SafeHome software engineering team.

The conversation:

Shakira (looking in through the entrance to the cubical): Hey . . . where were you at lunchtime?

Ed: Right here . . . working.

Shakira: You look miserable . . . what’s the matter?

Ed (sighing audibly): I’ve been working on this . . . bug since I discovered it at 9:30 this morning and it’s what, 2:45 . . . I’m clueless.

Shakira: I thought we all agreed to spend no more than one hour debugging stuff on our own; then we get help, right?

Ed: Yeah, but . . .

Shakira (walking into the cubical): So what’s the problem?

Ed: It’s complicated, and besides, I’ve been looking at this for, what, 5 hours. You’re not going to see it in 5 minutes.

Shakira: Indulge me . . . what’s the problem? [Ed explains the problem to Shakira, who looks at it for

about 30 seconds without speaking, then . . .]

Shakira (a smile is gathering on her face): Uh, right there, the variable named setAlarmCondition. Shouldn’t it be set to “false” before the loop gets started?

[Ed stares at the screen in disbelief, bends forward, and begins to bang his head gently against the monitor. Shakira, smiling broadly now, stands and walks out.]

SAFEHOME

17.8.3 Debugging Strategies

Regardless of the approach that is taken, debugging has one overriding objective— to find and correct the cause of a software error or defect. The objective is realized by a combination of systematic evaluation, intuition, and luck. Bradley [Bra85] describes the debugging approach in this way:

Debugging is a straightforward application of the scientific method that has been devel-

oped over 2,500 years. The basis of debugging is to locate the problem’s source [the

cause] by binary partitioning, through working hypotheses that predict new values to be

examined.

Take a simple non-software example: A lamp in my house does not work. If nothing

in the house works, the cause must be in the main circuit breaker or outside; I look around

pre75977_ch17.qxd 11/27/08 6:09 PM Page 475

to see whether the neighborhood is blacked out. I plug the suspect lamp into a working

socket and a working appliance into the suspect circuit. So goes the alternation of

hypothesis and test.

In general, three debugging strategies have been proposed [Mye79]: (1) brute force, (2) backtracking, and (3) cause elimination. Each of these strategies can be conducted manually, but modern debugging tools can make the process much more effective.

Debugging tactics. The brute force category of debugging is probably the most common and least efficient method for isolating the cause of a software error. You apply brute force debugging methods when all else fails. Using a “let the computer find the error” philosophy, memory dumps are taken, run-time traces are invoked, and the program is loaded with output statements. You hope that somewhere in the morass of information that is produced you’ll find a clue that can lead to the cause of an error. Although the mass of information produced may ultimately lead to success, it more frequently leads to wasted effort and time. Thought must be expended first!

Backtracking is a fairly common debugging approach that can be used success- fully in small programs. Beginning at the site where a symptom has been uncovered, the source code is traced backward (manually) until the cause is found. Unfortu- nately, as the number of source lines increases, the number of potential backward paths may become unmanageably large.

The third approach to debugging—cause elimination—is manifested by induction or deduction and introduces the concept of binary partitioning. Data related to the error occurrence are organized to isolate potential causes. A “cause hypothesis” is devised and the aforementioned data are used to prove or disprove the hypothesis. Alternatively, a list of all possible causes is developed and tests are conducted to eliminate each. If initial tests indicate that a particular cause hypothesis shows promise, data are refined in an attempt to isolate the bug.

Automated debugging. Each of these debugging approaches can be supple- mented with debugging tools that can provide you with semiautomated support as debugging strategies are attempted. Hailpern and Santhanam [Hai02] summarize the state of these tools when they note, “. . . many new approaches have been pro- posed and many commercial debugging environments are available. Integrated development environments (IDEs) provide a way to capture some of the language- specific predetermined errors (e.g., missing end-of-statement characters, unde- fined variables, and so on) without requiring compilation.” A wide variety of debugging compilers, dynamic debugging aids (“tracers”), automatic test-case generators, and cross-reference mapping tools are available. However, tools are not a substitute for careful evaluation based on a complete design model and clear source code.

476 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

uote:

“The first step in fixing a broken program is getting it to fail repeatably (on the simplest example possible).”

T. Duff

pre75977_ch17.qxd 11/27/08 6:09 PM Page 476

The people factor. Any discussion of debugging approaches and tools is incom- plete without mention of a powerful ally—other people! A fresh viewpoint, un- clouded by hours of frustration, can do wonders.7 A final maxim for debugging might be: “When all else fails, get help!”

17.8.4 Correcting the Error

Once a bug has been found, it must be corrected. But, as we have already noted, the correction of a bug can introduce other errors and therefore do more harm than good. Van Vleck [Van89] suggests three simple questions that you should ask before making the “correction” that removes the cause of a bug:

1. Is the cause of the bug reproduced in another part of the program? In many situ- ations, a program defect is caused by an erroneous pattern of logic that may be reproduced elsewhere. Explicit consideration of the logical pattern may result in the discovery of other errors.

2. What “next bug” might be introduced by the fix I’m about to make? Before the correction is made, the source code (or, better, the design) should be evalu- ated to assess coupling of logic and data structures. If the correction is to be made in a highly coupled section of the program, special care must be taken when any change is made.

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 477

Debugging Objective: These tools provide automated assistance for those who must debug software

problems. The intent is to provide insight that may be difficult to obtain if approaching the debugging process manually.

Mechanics: Most debugging tools are programming language and environment specific.

Representative Tools:6

Borland Gauntlet, distributed by Borland (www.borland.com), assists in both testing and debugging.

Coverty Prevent SQS, developed by Coverty (www.coverty.com), provides debugging assistance for both C++ and Java.

C++Test, developed by Parasoft (www.parasoft.com), is a unit-testing tool that supports a full range of tests on C and C++ code. Debugging features assist in the diagnosis of errors that are found.

CodeMedic, developed by NewPlanet Software (www.newplanetsoftware.com/medic/), provides a graphical interface for the standard UNIX debugger, gdb, and implements its most important features. gdb currently supports C/C++, Java, PalmOS, various embedded systems, assembly language, FORTRAN, and Modula-2.

GNATS, a freeware application (www.gnu.org/ software/gnats/), is a set of tools for tracking bug reports.

SOFTWARE TOOLS

6 Tools noted here do not represent an endorsement, but rather a sampling of tools in this category. In most cases, tool names are trademarked by their respective developers.

7 The concept of pair programming (recommended as part of the Extreme Programming model dis- cussed in Chapter 3) provides a mechanism for “debugging” as the software is designed and coded.

uote:

“The best tester isn’t the one who finds the most bugs … the best tester is the one who gets the most bugs fixed.”

Cem Kaner et al.

pre75977_ch17.qxd 11/27/08 6:09 PM Page 477

3. What could we have done to prevent this bug in the first place? This question is the first step toward establishing a statistical software quality assurance ap- proach (Chapter 16). If you correct the process as well as the product, the bug will be removed from the current program and may be eliminated from all future programs.

1 7 . 9 S U M M A R Y Software testing accounts for the largest percentage of technical effort in the soft- ware process. Regardless of the type of software you build, a strategy for systematic test planning, execution, and control begins by considering small elements of the software and moves outward toward the program as a whole.

The objective of software testing is to uncover errors. For conventional software, this objective is achieved through a series of test steps. Unit and integration tests concentrate on functional verification of a component and incorporation of compo- nents into the software architecture. Validation testing demonstrates traceability to software requirements, and system testing validates software once it has been incorporated into a larger system. Each test step is accomplished through a series of systematic test techniques that assist in the design of test cases. With each testing step, the level of abstraction with which software is considered is broadened.

The strategy for testing object-oriented software begins with tests that exercise the operations within a class and then moves to thread-based testing for integration. Threads are sets of classes that respond to an input or event. Use-based tests focus on classes that do not collaborate heavily with other classes.

WebApps are tested in much the same way as OO systems. However, tests are designed to exercise content, functionality, the interface, navigation, and aspects of WebApp performance and security.

Unlike testing (a systematic, planned activity), debugging can be viewed as an art. Beginning with a symptomatic indication of a problem, the debugging activity must track down the cause of an error. Of the many resources available during debugging, the most valuable is the counsel of other members of the software engineering staff.

P R O B L E M S A N D P O I N T S T O P O N D E R 17.1. Using your own words, describe the difference between verification and validation. Do both make use of test-case design methods and testing strategies?

17.2. List some problems that might be associated with the creation of an independent test group. Are an ITG and an SQA group made up of the same people?

17.3. Is it always possible to develop a strategy for testing software that uses the sequence of testing steps described in Section 17.1.3? What possible complications might arise for embed- ded systems?

17.4. Why is a highly coupled module difficult to unit test?

478 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

pre75977_ch17.qxd 11/27/08 6:09 PM Page 478

17.5. The concept of “antibugging” (Section 17.2.1) is an extremely effective way to provide built-in debugging assistance when an error is uncovered:

a. Develop a set of guidelines for antibugging. b. Discuss advantages of using the technique. c. Discuss disadvantages.

17.6. How can project scheduling affect integration testing?

17.7. Is unit testing possible or even desirable in all circumstances? Provide examples to justify your answer.

17.8. Who should perform the validation test—the software developer or the software user? Justify your answer.

17.9. Develop a complete test strategy for the SafeHome system discussed earlier in this book. Document it in a Test Specification.

17.10. As a class project, develop a Debugging Guide for your installation. The guide should provide language and system-oriented hints that have been learned through the school of hard knocks! Begin with an outline of topics that will be reviewed by the class and your instructor. Publish the guide for others in your local environment.

F U R T H E R R E A D I N G S A N D I N F O R M AT I O N S O U R C E S Virtually every book on software testing discusses strategies along with methods for test-case design. Everett and Raymond (Software Testing, Wiley-IEEE Computer Society Press, 2007), Black (Pragmatic Software Testing, Wiley, 2007), Spiller and his colleagues (Software Testing Process: Test Management, Rocky Nook, 2007), Perry (Effective Methods for Software Testing, 3d ed., Wiley, 2005), Lewis (Software Testing and Continuous Quality Improvement, 2d ed., Auerbach, 2004), Loveland and his colleagues (Software Testing Techniques, Charles River Media, 2004), Burnstein (Practical Software Testing, Springer, 2003), Dustin (Effective Software Testing, Addison-Wesley, 2002), Craig and Kaskiel (Systematic Software Testing, Artech House, 2002), Tamres (Introducing Software Test- ing, Addison-Wesley, 2002), Whittaker (How to Break Software, Addison-Wesley, 2002), and Kaner and his colleagues (Lessons Learned in Software Testing, Wiley, 2001) are only a small sampling of many books that discuss testing principles, concepts, strategies, and methods.

For those readers with interest in agile software development methods, Crispin and House (Testing Extreme Programming, Addison-Wesley, 2002) and Beck (Test Driven Development: By Example, Addison-Wesley, 2002) present testing strategies and tactics for Extreme Program- ming. Kamer and his colleagues (Lessons Learned in Software Testing, Wiley, 2001) present a collection of over 300 pragmatic “lessons” (guidelines) that every software tester should learn. Watkins (Testing IT: An Off-the-Shelf Testing Process, Cambridge University Press, 2001) estab- lishes an effective testing framework for all types of developed and acquired software. Manges and O’Brien (Agile Testing with Ruby and Rails, Apress, 2008) addresses testing strategies and techniques for the Ruby programming language and Web framework.

Sykes and McGregor (Practical Guide to Testing Object-Oriented Software, Addison-Wesley, 2001), Bashir and Goel (Testing Object-Oriented Software, Springer-Verlag, 2000), Binder (Testing Object-Oriented Systems, Addison-Wesley, 1999), Kung and his colleagues (Testing Object- Oriented Software, IEEE Computer Society Press, 1998), and Marick (The Craft of Software Testing, Prentice-Hall, 1997) present strategies and methods for testing OO systems.

Guidelines for debugging are contained in books by Grötker and his colleagues (The Devel- oper’s Guide to Debugging, Springer, 2008), Agans (Debugging, Amacon, 2006), Zeller (Why Programs Fail: A Guide to Systematic Debugging, Morgan Kaufmann, 2005), Tells and Hsieh (The Science of Debugging, The Coreolis Group, 2001), and Robbins (Debugging Applications, Microsoft Press, 2000). Kaspersky (Hacker Debugging Uncovered, A-List Publishing, 2005) addresses the technology of debugging tools. Younessi (Object-Oriented Defect Management of Software, Prentice-Hall, 2002) presents techniques for managing defects that are encountered in

C H A P T E R 1 7 S O F T W A R E T E S T I N G S T R A T E G I E S 479

pre75977_ch17.qxd 11/27/08 6:09 PM Page 479

object-oriented systems. Beizer [Bei84] presents an interesting “taxonomy of bugs” that can lead to effective methods for test planning.

Books by Madisetti and Akgul (Debugging Embedded Systems, Springer, 2007), Robbins (Debugging Microsoft .NET 2.0 Applications, Microsoft Press, 2005), Best (Linux Debugging and Performance Tuning, Prentice-Hall, 2005), Ford and Teorey (Practical Debugging in C++, Prentice- Hall, 2002), Brown (Debugging Perl, McGraw-Hill, 2000), and Mitchell (Debugging Java, McGraw- Hill, 2000) address the special nature of debugging for the environments implied by their titles.

A wide variety of information sources on software testing strategies are available on the Internet. An up-to-date list of World Wide Web references that are relevant to software testing strategies can be found at the SEPA website: www.mhhe.com/engcs/compsci/pressman/ professional/olc/ser.htm.

480 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

pre75977_ch17.qxd 11/27/08 6:09 PM Page 480