short answers (250 words) + Discussion (150 words)

Testing presents an interesting anomaly for software engineers, who by theirnature are constructive people. Testing requires that the developer discardpreconceived notions of the “correctness” of software just developed and then work hard to design test cases to “break” the software. Beizer [Bei90] describes this situation effectively when he states:

There’s a myth that if we were really good at programming, there would be no bugs to

catch. If only we could really concentrate, if only everyone used structured program-

ming, top-down design, . . . then there would be no bugs. So goes the myth. There are

bugs, the myth says, because we are bad at what we do; and if we are bad at it, we

should feel guilty about it. Therefore, testing and test case design is an admission of

failure, which instills a goodly dose of guilt. And the tedium of testing is just punish-

ment for our errors. Punishment for what? For being human? Guilt for what? For fail-

ing to achieve inhuman perfection? For not distinguishing between what another

programmer thinks and what he says? For failing to be telepathic? For not solving

human communications problems that have been kicked around . . . for forty centuries?

Should testing instill guilt? Is testing really destructive? The answer to these ques- tions is “No!”

In this chapter, I discuss techniques for software test-case design for conven- tional applications. Test-case design focuses on a set of techniques for the cre- ation of test cases that meet overall testing objectives and the testing strategies discussed in Chapter 17.

481

C H A P T E R

18TESTING CONVENTIONALAPPLICATIONS

What is it? Once source code has been generated, software must be tested to uncover (and correct) as many errors as possible before deliv-

ery to your customer. Your goal is to design a series of test cases that have a high likelihood of finding errors—but how? That’s where software testing techniques enter the picture. These tech- niques provide systematic guidance for design- ing tests that (1) exercise the internal logic and interfaces of every software component and (2) exercise the input and output domains of the pro- gram to uncover errors in program function, behavior, and performance.

Q U I C K L O O K

Who does it? During early stages of testing, a software engineer performs all tests. However, as the testing process progresses, testing spe- cialists may become involved.

Why is it important? Reviews and other SQA actions can and do uncover errors, but they are not sufficient. Every time the program is executed, the customer tests it! Therefore, you have to exe- cute the program before it gets to the customer with the specific intent of finding and removing all errors. In order to find the highest possible num- ber of errors, tests must be conducted systemati- cally and test cases must be designed using disciplined techniques.

K E Y C O N C E P T S basis path testing . . . . . . . . .485 black-box testing . . . . . . . . .495 boundary value analysis . . . . . . . .498 control structure testing . . . . . . . . .492 cyclomatic complexity . . . . . .488 equivalence partitioning . . . . . .497 flow graph . . . . . .485 graph-based testing methods . . . . . . . .495 graph matrices . . .491 model-based testing . . . . . . . . .502 orthogonal array testing . . . . . . . . .499 patterns . . . . . . . .507 specialized environments . . . .503 white-box testing . . . . . . . . .485

pre75977_ch18.qxd 11/27/08 6:12 PM Page 481

482 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

1 8 . 1 S O F T WA R E T E S T I N G F U N D A M E N TA L S The goal of testing is to find errors, and a good test is one that has a high probabil- ity of finding an error. Therefore, you should design and implement a computer- based system or a product with “testability” in mind. At the same time, the tests themselves must exhibit a set of characteristics that achieve the goal of finding the most errors with a minimum of effort.

Testability. James Bach1 provides the following definition for testability: “Software testability is simply how easily [a computer program] can be tested.” The following characteristics lead to testable software.

Operability. “The better it works, the more efficiently it can be tested.” If a system is designed and implemented with quality in mind, relatively few bugs will block the execution of tests, allowing testing to progress without fits and starts.

Observability. “What you see is what you test.” Inputs provided as part of testing produce distinct outputs. System states and variables are visible or queriable dur- ing execution. Incorrect output is easily identified. Internal errors are automatically detected and reported. Source code is accessible.

Controllability. “The better we can control the software, the more the testing can be automated and optimized.” All possible outputs can be generated through some combination of input, and I/O formats are consistent and structured. All code is executable through some combination of input. Software and hardware states and

uote:

“Every program does something right, it just may not be the thing we want it to do.”

Author unknown

What are the character-

istics of testability?

?

1 The paragraphs that follow are used with permission of James Bach (copyright 1994) and have been adapted from material that originally appeared in a posting in the newsgroup comp.software-eng.

What are the steps? For conventional applica- tions, software is tested from two different per- spectives: (1) internal program logic is exercised using “white box” test-case design techniques and (2) software requirements are exercised using “black box” test-case design techniques. Use cases assist in the design of tests to uncover errors at the software validation level. In every case, the intent is to find the maximum number of errors with the minimum amount of effort and time.

What is the work product? A set of test cases designed to exercise both internal logic,

interfaces, component collaborations, and external requirements is designed and docu- mented, expected results are defined, and actual results are recorded.

How do I ensure that I’ve done it right? When you begin testing, change your point of view. Try hard to “break” the software! Design test cases in a disciplined fashion and review the test cases you do create for thoroughness. In addition, you can evaluate test coverage and track error detection activities.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 482

variables can be controlled directly by the test engineer. Tests can be conveniently specified, automated, and reproduced.

Decomposability. “By controlling the scope of testing, we can more quickly isolate problems and perform smarter retesting.” The software system is built from independent modules that can be tested independently.

Simplicity. “The less there is to test, the more quickly we can test it.” The pro- gram should exhibit functional simplicity (e.g., the feature set is the minimum nec- essary to meet requirements); structural simplicity (e.g., architecture is modularized to limit the propagation of faults), and code simplicity (e.g., a coding standard is adopted for ease of inspection and maintenance).

Stability. “The fewer the changes, the fewer the disruptions to testing.” Changes to the software are infrequent, controlled when they do occur, and do not invali- date existing tests. The software recovers well from failures.

Understandability. “The more information we have, the smarter we will test.” The architectural design and the dependencies between internal, external, and shared components are well understood. Technical documentation is instantly accessible, well organized, specific and detailed, and accurate. Changes to the design are communicated to testers.

You can use the attributes suggested by Bach to develop a software configuration (i.e., programs, data, and documents) that is amenable to testing.

Test Characteristics. And what about the tests themselves? Kaner, Falk, and Nguyen [Kan93] suggest the following attributes of a “good” test:

A good test has a high probability of finding an error. To achieve this goal, the tester must understand the software and attempt to develop a mental picture of how the software might fail. Ideally, the classes of failure are probed. For example, one class of potential failure in a graphical user interface is the failure to recognize proper mouse position. A set of tests would be designed to exercise the mouse in an attempt to demonstrate an error in mouse position recognition.

A good test is not redundant. Testing time and resources are limited. There is no point in conducting a test that has the same purpose as another test. Every test should have a different purpose (even if it is subtly different).

A good test should be “best of breed” [Kan93]. In a group of tests that have a simi- lar intent, time and resource limitations may mitigate toward the execution of only a subset of these tests. In such cases, the test that has the highest likelihood of uncovering a whole class of errors should be used.

A good test should be neither too simple nor too complex. Although it is sometimes possible to combine a series of tests into one test case, the possible side effects associated with this approach may mask errors. In general, each test should be executed separately.

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 483

uote:

“Errors are more common, more pervasive, and more troublesome in software than with other technologies.”

David Parnas

What is a “good”

test? ?

pre75977_ch18.qxd 11/27/08 6:12 PM Page 483

1 8 . 2 I N T E R N A L A N D E X T E R N A L V I E W S O F T E S T I N G Any engineered product (and most other things) can be tested in one of two ways: (1) Knowing the specified function that a product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function. (2) Knowing the internal workings of a prod- uct, tests can be conducted to ensure that “all gears mesh,” that is, internal operations are performed according to specifications and all internal components have been ade- quately exercised. The first test approach takes an external view and is called black-box testing. The second requires an internal view and is termed white-box testing.2

Black-box testing alludes to tests that are conducted at the software interface. A black-box test examines some fundamental aspect of a system with little regard for the internal logical structure of the software. White-box testing of software is pred- icated on close examination of procedural detail. Logical paths through the software and collaborations between components are tested by exercising specific sets of conditions and/or loops.

At first glance it would seem that very thorough white-box testing would lead to “100 percent correct programs.” All we need do is define all logical paths, develop test cases to exercise them, and evaluate results, that is, generate test cases to exercise program logic exhaustively. Unfortunately, exhaustive testing presents

484 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Designing Unique Tests

The scene: Vinod’s cubical.

The players: Vinod and Ed—members of the SafeHome software engineering team.

The conversation:

Vinod: So these are the test cases you intend to run for the passwordValidation operation.

Ed: Yeah, they should cover pretty much all possibilities for the kinds of passwords a user might enter.

Vinod: So let’s see . . . you note that the correct password will be 8080, right?

Ed: Uh huh.

Vinod: And you specify passwords 1234 and 6789 to test for error in recognizing invalid passwords?

Ed: Right, and I also test passwords that are close to the correct password, see . . . 8081 and 8180.

Vinod: Those are okay, but I don’t see much point in running both the 1234 and 6789 inputs. They’re redundant . . . test the same thing, don’t they?

Ed: Well, they’re different values.

Vinod: That’s true, but if 1234 doesn’t uncover an error . . . in other words . . . the passwordValidation operation notes that it’s an invalid password, it’s not likely that 6789 will show us anything new.

Ed: I see what you mean.

Vinod: I’m not trying to be picky here . . . it’s just that we have limited time to do testing, so it’s a good idea to run tests that have a high likelihood of finding new errors.

Ed: Not a problem . . . I’ll give this a bit more thought.

SAFEHOME

2 The terms functional testing and structural testing are sometimes used in place of black-box and white-box testing, respectively.

uote:

“There is only one rule in designing test cases: cover all features, but do not make too many test cases.”

Tsuneo Yamaura

White-box tests can be designed only after component-level design (or source code) exists. The logical details of the program must be available.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 484

certain logistical problems. For even small programs, the number of possible logical paths can be very large. White-box testing should not, however, be dismissed as impractical. A limited number of important logical paths can be selected and exercised. Important data structures can be probed for validity.

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 485

Exhaustive Testing Consider a 100-line program in the language C. After some basic data declaration, the

program contains two nested loops that execute from 1 to 20 times each, depending on conditions specified at input. Inside the interior loop, four if-then-else constructs are required. There are approximately 1014 possible paths that may be executed in this program!

To put this number in perspective, we assume that a magic test processor (“magic” because no such processor

exists) has been developed for exhaustive testing. The processor can develop a test case, execute it, and evaluate the results in one millisecond. Working 24 hours a day, 365 days a year, the processor would work for 3170 years to test the program. This would, undeniably, cause havoc in most development schedules.

Therefore, it is reasonable to assert that exhaustive testing is impossible for large software systems.

INFO

1 8 . 3 W H I T E - B O X T E S T I N G White-box testing, sometimes called glass-box testing, is a test-case design philoso- phy that uses the control structure described as part of component-level design to derive test cases. Using white-box testing methods, you can derive test cases that (1) guarantee that all independent paths within a module have been exercised at least once, (2) exercise all logical decisions on their true and false sides, (3) execute all loops at their boundaries and within their operational bounds, and (4) exercise internal data structures to ensure their validity.

1 8 . 4 B A S I S PAT H T E S T I N G Basis path testing is a white-box testing technique first proposed by Tom McCabe [McC76]. The basis path method enables the test-case designer to derive a logical complexity measure of a procedural design and use this measure as a guide for defin- ing a basis set of execution paths. Test cases derived to exercise the basis set are guar- anteed to execute every statement in the program at least one time during testing.

18.4.1 Flow Graph Notation

Before we consider the basis path method, a simple notation for the representation of control flow, called a flow graph (or program graph) must be introduced.3 The flow graph depicts logical control flow using the notation illustrated in Figure 18.1. Each structured construct (Chapter 10) has a corresponding flow graph symbol.

uote:

“Bugs lurk in corners and congregate at boundaries.”

Boris Beizer

3 In actuality, the basis path method can be conducted without the use of flow graphs. However, they serve as a useful notation for understanding control flow and illustrating the approach.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 485

To illustrate the use of a flow graph, consider the procedural design representation in Figure 18.2a. Here, a flowchart is used to depict program control structure. Figure 18.2b maps the flowchart into a corresponding flow graph (assuming that no compound conditions are contained in the decision diamonds of the flowchart). Referring to Figure 18.2b, each circle, called a flow graph node, represents one or more procedural statements. A sequence of process boxes and a decision diamond can map into a sin- gle node. The arrows on the flow graph, called edges or links, represent flow of con- trol and are analogous to flowchart arrows. An edge must terminate at a node, even if the node does not represent any procedural statements (e.g., see the flow graph symbol for the if-then-else construct). Areas bounded by edges and nodes are called regions. When counting regions, we include the area outside the graph as a region.4

486 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

If While

The structured constructs in flow graph form:

Where each circle represents one or more nonbranching PDL or source code statements

Until

Case

Sequence

FIGURE 18.1 Flow graph notation

1

3

10

(a)

6

9

2

4

587

11 (b)

1

2,3

4,56

9

10

11

87

R1

R3

R2

R4

Region

Node

Edge

FIGURE 18.2 (a) Flowchart and (b) flow graph

4 A more detailed discussion of graphs and their uses is presented in Section 18.6.1.

A flow graph should be drawn only when the logical structure of a component is complex. The flow graph allows you to trace program paths more readily.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 486

When compound conditions are encountered in a procedural design, the genera- tion of a flow graph becomes slightly more complicated. A compound condition occurs when one or more Boolean operators (logical OR, AND, NAND, NOR) is pres- ent in a conditional statement. Referring to Figure 18.3, the program design language (PDL) segment translates into the flow graph shown. Note that a separate node is created for each of the conditions a and b in the statement IF a OR b. Each node that contains a condition is called a predicate node and is characterized by two or more edges emanating from it.

18.4.2 Independent Program Paths

An independent path is any path through the program that introduces at least one new set of processing statements or a new condition. When stated in terms of a flow graph, an independent path must move along at least one edge that has not been traversed before the path is defined. For example, a set of independent paths for the flow graph illustrated in Figure 18.2b is

Path 1: 1-11

Path 2: 1-2-3-4-5-10-1-11

Path 3: 1-2-3-6-8-9-10-1-11

Path 4: 1-2-3-6-7-9-10-1-11

Note that each new path introduces a new edge. The path

1-2-3-4-5-10-1-2-3-6-8-9-10-1-11

is not considered to be an independent path because it is simply a combination of already specified paths and does not traverse any new edges.

Paths 1 through 4 constitute a basis set for the flow graph in Figure 18.2b. That is, if you can design tests to force execution of these paths (a basis set), every statement in the program will have been guaranteed to be executed at least one time and every condition will have been executed on its true and false sides. It should be noted that

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 487

Predicate node

.

.

. IF a OR b

then procedure x else procedure y

ENDIF

y

b

a

x

x

FIGURE 18.3 Compound logic

pre75977_ch18.qxd 11/27/08 6:12 PM Page 487

the basis set is not unique. In fact, a number of different basis sets can be derived for a given procedural design.

How do you know how many paths to look for? The computation of cyclomatic complexity provides the answer. Cyclomatic complexity is a software metric that pro- vides a quantitative measure of the logical complexity of a program. When used in the context of the basis path testing method, the value computed for cyclomatic com- plexity defines the number of independent paths in the basis set of a program and provides you with an upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once.

Cyclomatic complexity has a foundation in graph theory and provides you with an extremely useful software metric. Complexity is computed in one of three ways:

1. The number of regions of the flow graph corresponds to the cyclomatic complexity.

2. Cyclomatic complexity V(G) for a flow graph G is defined as

V(G) ! E " N # 2

where E is the number of flow graph edges and N is the number of flow graph nodes.

3. Cyclomatic complexity V(G) for a flow graph G is also defined as

V(G) ! P # 1

where P is the number of predicate nodes contained in the flow graph G.

Referring once more to the flow graph in Figure 18.2b, the cyclomatic complexity can be computed using each of the algorithms just noted:

1. The flow graph has four regions.

2. V(G) ! 11 edges " 9 nodes # 2 ! 4.

3. V(G) ! 3 predicate nodes # 1 ! 4.

Therefore, the cyclomatic complexity of the flow graph in Figure 18.2b is 4. More important, the value for V(G) provides you with an upper bound for the num-

ber of independent paths that form the basis set and, by implication, an upper bound on the number of tests that must be designed and executed to guarantee coverage of all program statements.

488 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Cyclomatic complexity is a useful metric for predicting those modules that are likely to be error prone. Use it for test planning as well as test-case design.

How do I compute

cyclomatic complexity?

?

Cyclomatic complexity provides the upper bound on the number of test cases that will be required to guarantee that every statement in the program has been executed at least one time.

The scene: Shakira’s cubicle.

The players: Vinod and Shakira—members of the SafeHome software engineering team who are working on test planning for the security function.

The conversation:

Shakira: Look . . . I know that we should unit-test all the components for the security function, but there are a lot of ‘em and if you consider the number of operations that

SAFEHOME

Using Cyclomatic Complexity

pre75977_ch18.qxd 11/27/08 6:12 PM Page 488

18.4.3 Deriving Test Cases

The basis path testing method can be applied to a procedural design or to source code. In this section, I present basis path testing as a series of steps. The procedure average, depicted in PDL in Figure 18.4, will be used as an example to illustrate each step in the test-case design method. Note that average, although an extremely sim- ple algorithm, contains compound conditions and loops. The following steps can be applied to derive the basis set:

1. Using the design or code as a foundation, draw a corresponding flow graph. A flow graph is created using the symbols and construction rules presented in Section 18.4.1. Referring to the PDL for average in Figure 18.4, a flow graph is created by numbering those PDL statements that will be mapped into corresponding flow graph nodes. The corresponding flow graph is shown in Figure 18.5.

2. Determine the cyclomatic complexity of the resultant flow graph. The cyclomatic complexity V(G) is determined by applying the algorithms described in Section 18.4.2. It should be noted that V(G) can be determined without developing a flow graph by counting all conditional statements in the PDL (for the procedure average, compound conditions count as two) and adding 1. Referring to Figure 18.5,

V(G) ! 6 regions V(G) ! 17 edges " 13 nodes # 2 ! 6 V(G) ! 5 predicate nodes # 1 ! 6

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 489

have to be exercised, I don’t know . . . maybe we should forget white-box testing, integrate everything, and start running black-box tests.

Vinod: You figure we don’t have enough time to do component tests, exercise the operations, and then integrate?

Shakira: The deadline for the first increment is getting closer than I’d like . . . yeah, I’m concerned.

Vinod: Why don’t you at least run white-box tests on the operations that are likely to be the most error prone?

Shakira (exasperated): And exactly how do I know which are the most error prone?

Vinod: V of G.

Shakira: Huh?

Vinod: Cyclomatic complexity—V of G. Just compute V(G) for each of the operations within each of the

components and see which have the highest values for V(G). They’re the ones that are most likely to be error prone.

Shakira: And how do I compute V of G?

Vinod: It’s really easy. Here’s a book that describes how to do it.

Shakira (leafing through the pages): Okay, it doesn’t look hard. I’ll give it a try. The ops with the highest V(G) will be the candidates for white-box tests.

Vinod: Just remember that there are no guarantees. A component with a low V(G) can still be error prone.

Shakira: Alright. But at least this’ll help me to narrow down the number of components that have to undergo white-box testing.

uote:

“The Ariane 5 rocket blew up on lift-off due solely to a software defect (a bug) involving the conversion of a 64- bit floating point value into a 16-bit integer. The rocket and its four satellites were uninsured and worth $500 million. [Path tests that exercised the conversion path] would have found the bug but were vetoed for budgetary reasons.”

A news report

pre75977_ch18.qxd 11/27/08 6:12 PM Page 489

490 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

PROCEDURE average;

INTERFACE RETURNS average, total.input, total.valid; INTERFACE ACCEPTS value, minimum, maximum;

TYPE value[1:100] IS SCALAR ARRAY; TYPE average, total.input, total.valid; minimum, maximum, sum IS SCALAR; TYPE i IS INTEGER;

* This procedure computes the average of 100 or fewer numbers that lie between bounding values; it also computes the sum and the total number valid.

i = 1; total.input = total.valid = 0; sum = 0; DO WHILE value[i] <> –999 AND total.input < 100

ENDDO IF total.valid > 0

ENDIF END average

increment total.input by 1; IF value[i] > = minimum AND value[i] < = maximum

ENDIF increment i by 1;

THEN average = sum / total.valid; ELSE average = –999;

THEN increment total.valid by 1; sum = s sum + value[i] ELSE skip

1

3

6 4

5 7

8

9 10

11 12

13

2

FIGURE 18.4 PDL with nodes identified

1

2

3

4

5

6

7

8

9

10

1112

13

FIGURE 18.5 Flow graph for the procedure average

3. Determine a basis set of linearly independent paths. The value of V(G) provides the upper bound on the number of linearly independent paths through the program control structure. In the case of procedure average, we expect to specify six paths:

Path 1: 1-2-10-11-13

Path 2: 1-2-10-12-13

pre75977_ch18.qxd 11/27/08 6:12 PM Page 490

Path 3: 1-2-3-10-11-13

Path 4: 1-2-3-4-5-8-9-2-. . .

Path 5: 1-2-3-4-5-6-8-9-2-. . .

Path 6: 1-2-3-4-5-6-7-8-9-2-. . .

The ellipsis (. . .) following paths 4, 5, and 6 indicates that any path through the remainder of the control structure is acceptable. It is often worthwhile to identify predicate nodes as an aid in the derivation of test cases. In this case, nodes 2, 3, 5, 6, and 10 are predicate nodes.

4. Prepare test cases that will force execution of each path in the basis set. Data should be chosen so that conditions at the predicate nodes are appropriately set as each path is tested. Each test case is executed and com- pared to expected results. Once all test cases have been completed, the tester can be sure that all statements in the program have been executed at least once.

It is important to note that some independent paths (e.g., path 1 in our example) cannot be tested in stand-alone fashion. That is, the combination of data required to traverse the path cannot be achieved in the normal flow of the program. In such cases, these paths are tested as part of another path test.

18.4.4 Graph Matrices

The procedure for deriving the flow graph and even determining a set of basis paths is amenable to mechanization. A data structure, called a graph matrix, can be quite useful for developing a software tool that assists in basis path testing.

A graph matrix is a square matrix whose size (i.e., number of rows and columns) is equal to the number of nodes on the flow graph. Each row and column corre- sponds to an identified node, and matrix entries correspond to connections (an edge) between nodes. A simple example of a flow graph and its corresponding graph matrix [Bei90] is shown in Figure 18.6.

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 491

1

3

4

2

5

a

b

c

d e

f g

Flow graph

1 3 42 5

1

3

4

2

5

a

eg

c f

d b

Connected to node

Node

Graph matrix

FIGURE 18.6 Graph matrix

pre75977_ch18.qxd 11/27/08 6:12 PM Page 491

Referring to the figure, each node on the flow graph is identified by numbers, while each edge is identified by letters. A letter entry is made in the matrix to correspond to a connection between two nodes. For example, node 3 is connected to node 4 by edge b.

To this point, the graph matrix is nothing more than a tabular representation of a flow graph. However, by adding a link weight to each matrix entry, the graph matrix can become a powerful tool for evaluating program control structure during testing. The link weight provides additional information about control flow. In its simplest form, the link weight is 1 (a connection exists) or 0 (a connection does not exist). But link weights can be assigned other, more interesting properties:

• The probability that a link (edge) will be execute.

• The processing time expended during traversal of a link

• The memory required during traversal of a link

• The resources required during traversal of a link.

Beizer [Bei90] provides a thorough treatment of additional mathematical algo- rithms that can be applied to graph matrices. Using these techniques, the analysis required to design test cases can be partially or fully automated.

1 8 . 5 C O N T R O L S T R U C T U R E T E S T I N G The basis path testing technique described in Section 18.4 is one of a number of tech- niques for control structure testing. Although basis path testing is simple and highly effective, it is not sufficient in itself. In this section, other variations on control struc- ture testing are discussed. These broaden testing coverage and improve the quality of white-box testing.

18.5.1 Condition Testing

Condition testing [Tai89] is a test-case design method that exercises the logical con- ditions contained in a program module. A simple condition is a Boolean variable or a relational expression, possibly preceded with one NOT (¬) operator. A relational expression takes the form

E1 <relational-operator> E2

where E1 and E2 are arithmetic expressions and <relational-operator> is one of the following: $, %, !, ! (nonequality), &, or '. A compound condition is composed of two or more simple conditions, Boolean operators, and parentheses. We assume that Boolean operators allowed in a compound condition include OR ( ! ), AND (&), and NOT (¬). A condition without relational expressions is referred to as a Boolean expression.

If a condition is incorrect, then at least one component of the condition is incor- rect. Therefore, types of errors in a condition include Boolean operator errors

492 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

What is a graph matrix

and how do I extend it for use in testing?

?

uote:

“Paying more attention to running tests than to designing them is a classic mistake.”

Brian Marick

Errors are much more common in the neighborhood of logical conditions than they are in the locus of sequential processing statements.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 492

(incorrect/missing/extra Boolean operators), Boolean variable errors, Boolean parenthesis errors, relational operator errors, and arithmetic expression errors. The condition testing method focuses on testing each condition in the program to ensure that it does not contain errors.

18.5.2 Data Flow Testing

The data flow testing method [Fra93] selects test paths of a program according to the locations of definitions and uses of variables in the program. To illustrate the data flow testing approach, assume that each statement in a program is assigned a unique statement number and that each function does not modify its parameters or global variables. For a statement with S as its statement number,

DEF(S) ! {X | statement S contains a definition of X} USE(S) ! {X | statement S contains a use of X}

If statement S is an if or loop statement, its DEF set is empty and its USE set is based on the condition of statement S. The definition of variable X at statement S is said to be live at statement S’ if there exists a path from statement S to statement S’ that con- tains no other definition of X.

A definition-use (DU) chain of variable X is of the form [X, S, S’], where S and S’ are statement numbers, X is in DEF(S) and USE(S’), and the definition of X in statement S is live at statement S’.

One simple data flow testing strategy is to require that every DU chain be covered at least once. We refer to this strategy as the DU testing strategy. It has been shown that DU testing does not guarantee the coverage of all branches of a program. How- ever, a branch is not guaranteed to be covered by DU testing only in rare situations such as if-then-else constructs in which the then part has no definition of any vari- able and the else part does not exist. In this situation, the else branch of the if state- ment is not necessarily covered by DU testing.

18.5.3 Loop Testing

Loops are the cornerstone for the vast majority of all algorithms implemented in software. And yet, we often pay them little heed while conducting software tests.

Loop testing is a white-box testing technique that focuses exclusively on the validity of loop constructs. Four different classes of loops [Bei90] can be defined: sim- ple loops, concatenated loops, nested loops, and unstructured loops (Figure 18.7).

Simple loops. The following set of tests can be applied to simple loops, where n is the maximum number of allowable passes through the loop.

1. Skip the loop entirely.

2. Only one pass through the loop.

3. Two passes through the loop.

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 493

It is unrealistic to assume that data flow testing will be used extensively when testing a large system. However, it can be used in a targeted fashion for areas of software that are suspect.

uote:

“Good testers are masters at noticing ‘something funny’ and acting on it.”

Brian Marick

pre75977_ch18.qxd 11/27/08 6:12 PM Page 493

4. m passes through the loop where m $ n.

5. n " 1, n, n # 1 passes through the loop.

Nested loops. If we were to extend the test approach for simple loops to nested loops, the number of possible tests would grow geometrically as the level of nesting increases. This would result in an impractical number of tests. Beizer [Bei90] sug- gests an approach that will help to reduce the number of tests:

1. Start at the innermost loop. Set all other loops to minimum values.

2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter (e.g., loop counter) values. Add other tests for out-of-range or excluded values.

3. Work outward, conducting tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical” values.

4. Continue until all loops have been tested.

Concatenated loops. Concatenated loops can be tested using the approach defined for simple loops, if each of the loops is independent of the other. However, if two loops are concatenated and the loop counter for loop 1 is used as the initial value for loop 2, then the loops are not independent. When the loops are not inde- pendent, the approach applied to nested loops is recommended.

Unstructured loops. Whenever possible, this class of loops should be redesigned to reflect the use of the structured programming constructs (Chapter 10).

494 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Simple loops Nested loops

Concatenated loops

Unstructured loops

FIGURE 18.7 Classes of Loops

You can’t test unstruc- tured loops effectively. Refactor them.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 494

1 8 . 6 B L A C K - B O X T E S T I N G Black-box testing, also called behavioral testing, focuses on the functional require- ments of the software. That is, black-box testing techniques enable you to derive sets of input conditions that will fully exercise all functional requirements for a program. Black-box testing is not an alternative to white-box techniques. Rather, it is a com- plementary approach that is likely to uncover a different class of errors than white- box methods.

Black-box testing attempts to find errors in the following categories: (1) incorrect or missing functions, (2) interface errors, (3) errors in data structures or external database access, (4) behavior or performance errors, and (5) initialization and termination errors.

Unlike white-box testing, which is performed early in the testing process, black- box testing tends to be applied during later stages of testing (see Chapter 17). Because black-box testing purposely disregards control structure, attention is focused on the information domain. Tests are designed to answer the following questions:

• How is functional validity tested?

• How are system behavior and performance tested?

• What classes of input will make good test cases?

• Is the system particularly sensitive to certain input values?

• How are the boundaries of a data class isolated?

• What data rates and data volume can the system tolerate?

• What effect will specific combinations of data have on system operation?

By applying black-box techniques, you derive a set of test cases that satisfy the fol- lowing criteria [Mye79]: (1) test cases that reduce, by a count that is greater than one, the number of additional test cases that must be designed to achieve reasonable testing, and (2) test cases that tell you something about the presence or absence of classes of errors, rather than an error associated only with the specific test at hand.

18.6.1 Graph-Based Testing Methods

The first step in black-box testing is to understand the objects5 that are modeled in software and the relationships that connect these objects. Once this has been accomplished, the next step is to define a series of tests that verify “all objects have the expected relationship to one another” [Bei95]. Stated in another way, software testing begins by creating a graph of important objects and their relationships and

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 495

What questions do

black-box tests answer?

?

A graph represents the relationships between data objects and program objects, enabling you to derive test cases that search for errors associated with these relationships.

5 In this context, you should consider the term objects in the broadest possible context. It encom- passes data objects, traditional components (modules), and object-oriented elements of computer software.

uote:

“To err is human, to find a bug is divine.”

Robert Dunn

pre75977_ch18.qxd 11/27/08 6:12 PM Page 495

then devising a series of tests that will cover the graph so that each object and rela- tionship is exercised and errors are uncovered.

To accomplish these steps, you begin by creating a graph—a collection of nodes that represent objects, links that represent the relationships between objects, node weights that describe the properties of a node (e.g., a specific data value or state behavior), and link weights that describe some characteristic of a link.

The symbolic representation of a graph is shown in Figure 18.8a. Nodes are represented as circles connected by links that take a number of different forms. A directed link (represented by an arrow) indicates that a relationship moves in only one direction. A bidirectional link, also called a symmetric link, implies that the rela- tionship applies in both directions. Parallel links are used when a number of different relationships are established between graph nodes.

As a simple example, consider a portion of a graph for a word-processing application (Figure 18.8b) where

Object #1 ! newFile (menu selection)

Object #2 ! documentWindow

Object #3 ! documentText

Referring to the figure, a menu select on newFile generates a document window. The node weight of documentWindow provides a list of the window attributes that are to be expected when the window is generated. The link weight indicates that the

496 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

New file menu select

Menu select generates Document window

Document text

Is represented as Contains

(b)

Object #1

Directed link Object #2

Object #3

Undirected link

Parallel links

Node weight (value)

(a)

Allows editing of

(link weight)

(generation time < 1.0 sec)

Attributes: Start dimension: default setting or preferences Background color: white Text color: default color or preferences

FIGURE 18.8 (a) Graph notation; (b) simple example

pre75977_ch18.qxd 11/27/08 6:12 PM Page 496

window must be generated in less than 1.0 second. An undirected link establishes a symmetric relationship between the newFile menu selection and documentText, and parallel links indicate relationships between documentWindow and documentText. In reality, a far more detailed graph would have to be generated as a precursor to test-case design. You can then derive test cases by traversing the graph and covering each of the relationships shown. These test cases are designed in an attempt to find errors in any of the relationships. Beizer [Bei95] describes a number of behavioral testing methods that can make use of graphs:

Transaction flow modeling. The nodes represent steps in some transac- tion (e.g., the steps required to make an airline reservation using an online service), and the links represent the logical connection between steps (e.g., flightInformationInput is followed by validationAvailabilityProcessing). The data flow diagram (Chapter 7) can be used to assist in creating graphs of this type.

Finite state modeling. The nodes represent different user-observable states of the software (e.g., each of the “screens” that appear as an order entry clerk takes a phone order), and the links represent the transitions that occur to move from state to state (e.g., orderInformation is verified during inventoryAvail- abilityLook-up and is followed by customerBillingInformation input). The state diagram (Chapter 7) can be used to assist in creating graphs of this type.

Data flow modeling. The nodes are data objects, and the links are the transformations that occur to translate one data object into another. For example, the node FICA tax withheld (FTW) is computed from gross wages (GW) using the relationship, FTW ! 0.62 " GW.

Timing modeling. The nodes are program objects, and the links are the sequential connections between those objects. Link weights are used to specify the required execution times as the program executes.

A detailed discussion of each of these graph-based testing methods is beyond the scope of this book. If you have further interest, see [Bei95] for a comprehensive coverage.

18.6.2 Equivalence Partitioning

Equivalence partitioning is a black-box testing method that divides the input domain of a program into classes of data from which test cases can be derived. An ideal test case single-handedly uncovers a class of errors (e.g., incorrect processing of all character data) that might otherwise require many test cases to be executed before the general error is observed.

Test-case design for equivalence partitioning is based on an evaluation of equivalence classes for an input condition. Using concepts introduced in the preced- ing section, if a set of objects can be linked by relationships that are symmetric,

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 497

Input classes are known relatively early in the software process. For this reason, begin thinking about equivalence partitioning as the design is created.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 497

transitive, and reflexive, an equivalence class is present [Bei95]. An equivalence class represents a set of valid or invalid states for input conditions. Typically, an input condition is either a specific numeric value, a range of values, a set of related values, or a Boolean condition. Equivalence classes may be defined according to the following guidelines:

1. If an input condition specifies a range, one valid and two invalid equivalence classes are defined.

2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined.

3. If an input condition specifies a member of a set, one valid and one invalid equivalence class are defined.

4. If an input condition is Boolean, one valid and one invalid class are defined.

By applying the guidelines for the derivation of equivalence classes, test cases for each input domain data item can be developed and executed. Test cases are selected so that the largest number of attributes of an equivalence class are exercised at once.

18.6.3 Boundary Value Analysis

A greater number of errors occurs at the boundaries of the input domain rather than in the “center.” It is for this reason that boundary value analysis (BVA) has been de- veloped as a testing technique. Boundary value analysis leads to a selection of test cases that exercise bounding values.

Boundary value analysis is a test-case design technique that complements equiv- alence partitioning. Rather than selecting any element of an equivalence class, BVA leads to the selection of test cases at the “edges” of the class. Rather than focusing solely on input conditions, BVA derives test cases from the output domain as well [Mye79].

Guidelines for BVA are similar in many respects to those provided for equivalence partitioning:

1. If an input condition specifies a range bounded by values a and b, test cases should be designed with values a and b and just above and just below a and b.

2. If an input condition specifies a number of values, test cases should be devel- oped that exercise the minimum and maximum numbers. Values just above and below minimum and maximum are also tested.

3. Apply guidelines 1 and 2 to output conditions. For example, assume that a tem- perature versus pressure table is required as output from an engineering analy- sis program. Test cases should be designed to create an output report that produces the maximum (and minimum) allowable number of table entries.

4. If internal program data structures have prescribed boundaries (e.g., a table has a defined limit of 100 entries), be certain to design a test case to exercise the data structure at its boundary.

498 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

How do I define

equivalence classes for testing?

?

uote:

“An effective way to test code is to exercise it at its natural boundaries.”

Brian Kernighan

BVA extends equivalence partitioning by focusing on data at the “edges” of an equivalence class.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 498

Most software engineers intuitively perform BVA to some degree. By applying these guidelines, boundary testing will be more complete, thereby having a higher likelihood for error detection.

18.6.4 Orthogonal Array Testing

There are many applications in which the input domain is relatively limited. That is, the number of input parameters is small and the values that each of the parameters may take are clearly bounded. When these numbers are very small (e.g., three input parameters taking on three discrete values each), it is possible to consider every input permutation and exhaustively test the input domain. However, as the number of input values grows and the number of discrete values for each data item increases, exhaustive testing becomes impractical or impossible.

Orthogonal array testing can be applied to problems in which the input domain is relatively small but too large to accommodate exhaustive testing. The orthogonal array testing method is particularly useful in finding region faults—an error category associated with faulty logic within a software component.

To illustrate the difference between orthogonal array testing and more conven- tional “one input item at a time” approaches, consider a system that has three input items, X, Y, and Z. Each of these input items has three discrete values associated with it. There are 33 ! 27 possible test cases. Phadke [Pha97] suggests a geometric view of the possible test cases associated with X, Y, and Z illustrated in Figure 18.9. Referring to the figure, one input item at a time may be varied in sequence along each input axis. This results in relatively limited coverage of the input domain (repre- sented by the left-hand cube in the figure).

When orthogonal array testing occurs, an L9 orthogonal array of test cases is created. The L9 orthogonal array has a “balancing property” [Pha97]. That is, test cases (represented by dark dots in the figure) are “dispersed uniformly throughout the test domain,” as illustrated in the right-hand cube in Figure 18.9. Test coverage across the input domain is more complete.

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 499

YY X X

ZZ

One input item at a time L9 orthogonal array

FIGURE 18.9 A geometric view of test cases Source: [Pha97]

Orthogonal array testing enables you to design test cases that provide maximum test coverage with a reasonable number of test cases.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 499

To illustrate the use of the L9 orthogonal array, consider the send function for a fax application. Four parameters, P1, P2, P3, and P4, are passed to the send function. Each takes on three discrete values. For example, P1 takes on values:

P1 ! 1, send it now P1 ! 2, send it one hour later P1 ! 3, send it after midnight

P2, P3, and P4 would also take on values of 1, 2, and 3, signifying other send functions.

If a “one input item at a time” testing strategy were chosen, the following sequence of tests (P1, P2, P3, P4) would be specified: (1, 1, 1, 1), (2, 1, 1, 1), (3, 1, 1, 1), (1, 2, 1, 1), (1, 3, 1, 1), (1, 1, 2, 1), (1, 1, 3, 1), (1, 1, 1, 2), and (1, 1, 1, 3). Phadke [Pha97] assesses these test cases by stating:

Such test cases are useful only when one is certain that these test parameters do not

interact. They can detect logic faults where a single parameter value makes the software

malfunction. These faults are called single mode faults. This method cannot detect logic

faults that cause malfunction when two or more parameters simultaneously take certain

values; that is, it cannot detect any interactions. Thus its ability to detect faults is limited.

Given the relatively small number of input parameters and discrete values, exhaustive testing is possible. The number of tests required is 34 ! 81, large but man- ageable. All faults associated with data item permutation would be found, but the effort required is relatively high.

The orthogonal array testing approach enables you to provide good test coverage with far fewer test cases than the exhaustive strategy. An L9 orthogonal array for the fax send function is illustrated in Figure 18.10.

500 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

Test case Test parameters

P1 P2 P3 P4

1

2

3

3

1

2

2

3

1

1

2

3

2

3

1

3

1

2

1

2

3

1

2

3

1

2

3

1

1

1

2

2

2

3

3

3

1

2

3

4

5

6

7

8

9

FIGURE 18.10 An L9 orthog- onal array

pre75977_ch18.qxd 11/27/08 6:12 PM Page 500

Phadke [Pha97] assesses the result of tests using the L9 orthogonal array in the following manner:

Detect and isolate all single mode faults. A single mode fault is a consistent prob-

lem with any level of any single parameter. For example, if all test cases of factor P1 ! 1

cause an error condition, it is a single mode failure. In this example tests 1, 2 and 3

[Figure 18.10] will show errors. By analyzing the information about which tests show

errors, one can identify which parameter values cause the fault. In this example, by not-

ing that tests 1, 2, and 3 cause an error, one can isolate [logical processing associated

with “send it now” (P1 ! 1)] as the source of the error. Such an isolation of fault is

important to fix the fault.

Detect all double mode faults. If there exists a consistent problem when specific

levels of two parameters occur together, it is called a double mode fault. Indeed, a double

mode fault is an indication of pairwise incompatibility or harmful interactions between

two test parameters.

Multimode faults. Orthogonal arrays [of the type shown] can assure the detection of

only single and double mode faults. However, many multimode faults are also detected

by these tests.

You can find a detailed discussion of orthogonal array testing in [Pha89].

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 501

Test-Case Design Objective: To assist the software team in developing a complete set of test cases for both

black-box and white-box testing.

Mechanics: These tools fall into two broad categories: static testing tools and dynamic testing tools. Three different types of static testing tools are used in the industry: code-based testing tools, specialized testing languages, and requirements-based testing tools. Code- based testing tools accept source code as input and perform a number of analyses that result in the generation of test cases. Specialized testing languages (e.g., ATLAS) enable a software engineer to write detailed test specifications that describe each test case and the logistics for its execution. Requirements-based testing tools isolate specific user requirements and suggest test cases (or classes of tests) that will exercise the requirements. Dynamic testing tools interact with an executing program, checking path coverage, testing assertions about the value of specific variables, and otherwise instrumenting the execution flow of the program.

Representative Tools:6

McCabeTest, developed by McCabe & Associates (www.mccabe.com), implements a variety of path testing techniques derived from an assessment of cyclomatic complexity and other software metrics.

TestWorks, developed by Software Research, Inc. (www.soft.com/Products), is a complete set of automated testing tools that assists in the design of tests cases for software developed in C/C++ and Java and provides support for regression testing.

T-VEC Test Generation System, developed by T-VEC Technologies (www.t-vec.com), is a tool set that supports unit, integration, and validation testing by assisting in the design of test cases using information contained in an OO requirements specification.

e-TEST Suite, developed by Empirix, Inc. (www.empirix .com), encompasses a complete set of tools for testing WebApps, including tools that assist test-case design and test planning.

SOFTWARE TOOLS

6 Tools noted here do not represent an endorsement, but rather a sampling of tools in this category. In most cases, tool names are trademarked by their respective developers.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 501

1 8 . 7 M O D E L - B A S E D T E S T I N G Model-based testing (MBT) is a black-box testing technique that uses information contained in the requirements model as the basis for the generation of test cases. In many cases, the model-based testing technique uses UML state diagrams, an ele- ment of the behavioral model (Chapter 7), as the basis for the design of test cases.7

The MBT technique requires five steps:

1. Analyze an existing behavioral model for the software or create one. Recall that a behavioral model indicates how software will respond to exter- nal events or stimuli. To create the model, you should perform the steps discussed in Chapter 7: (1) evaluate all use cases to fully understand the sequence of interaction within the system, (2) identify events that drive the interaction sequence and understand how these events relate to specific objects, (3) create a sequence for each use case, (4) build a UML state diagram for the system (e.g., see Figure 7.6), and (5) review the behavioral model to verify accuracy and consistency.

2. Traverse the behavioral model and specify the inputs that will force the software to make the transition from state to state. The inputs will trigger events that will cause the transition to occur.

3. Review the behavioral model and note the expected outputs as the software makes the transition from state to state. Recall that each state transition is triggered by an event and that as a consequence of the transition, some function is invoked and outputs are created. For each set of inputs (test cases) you specified in step 2, specify the expected outputs as they are characterized in the behavioral model. “A fundamental assumption of this testing is that there is some mechanism, a test oracle, that will deter- mine whether or not the results of a test execution are correct” [DAC03]. In essence, a test oracle establishes the basis for any determination of the cor- rectness of the output. In most cases, the oracle is the requirements model, but it could also be another document or application, data recorded else- where, or even a human expert.

4. Execute the test cases. Tests can be executed manually or a test script can be created and executed using a testing tool.

5. Compare actual and expected results and take corrective action as required.

MBT helps to uncover errors in software behavior, and as a consequence, it is extremely useful when testing event-driven applications.

502 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

uote:

“It’s hard enough to find an error in your code when you’re looking for it; it’s even harder when you’ve assumed your code is error-free.”

Steve McConnell

7 Model-based testing can also be used when software requirements are represented with decision tables, grammars, or Markov chains [DAC03].

pre75977_ch18.qxd 11/27/08 6:12 PM Page 502

1 8 . 8 T E S T I N G F O R S P E C I A L I Z E D E N V I R O N M E N T S , A R C H I T E C T U R E S , A N D A P P L I C AT I O N S Unique guidelines and approaches to testing are sometimes warranted when spe- cialized environments, architectures, and applications are considered. Although the testing techniques discussed earlier in this chapter and in Chapters 19 and 20 can often be adapted to specialized situations, it’s worth considering their unique needs individually.

18.8.1 Testing GUIs

Graphical user interfaces (GUIs) will present you with interesting testing challenges. Because reusable components are now a common part of GUI development envi- ronments, the creation of the user interface has become less time consuming and more precise (Chapter 11). But, at the same time, the complexity of GUIs has grown, leading to more difficulty in the design and execution of test cases.

Because many modern GUIs have the same look and feel, a series of standard tests can be derived. Finite-state modeling graphs may be used to derive a series of tests that address specific data and program objects that are relevant to the GUI. This model-based testing technique was discussed in Section 18.7.

Because of the large number of permutations associated with GUI operations, GUI testing should be approached using automated tools. A wide array of GUI testing tools has appeared on the market over the past few years.8

18.8.2 Testing of Client-Server Architectures

The distributed nature of client-server environments, the performance issues asso- ciated with transaction processing, the potential presence of a number of different hardware platforms, the complexities of network communication, the need to serv- ice multiple clients from a centralized (or in some cases, distributed) database, and the coordination requirements imposed on the server all combine to make testing of client-server architectures and the software that resides within them considerably more difficult than stand-alone applications. In fact, recent industry studies indicate a significant increase in testing time and cost when client-server environments are developed.

In general, the testing of client-server software occurs at three different levels: (1) Individual client applications are tested in a “disconnected” mode; the operation of the server and the underlying network are not considered. (2) The client software and associated server applications are tested in concert, but network operations are not explicitly exercised. (3) The complete client-server architecture, including net- work operation and performance, is tested.

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 503

8 Hundreds, if not thousands, of GUI testing tool resources can be evaluated on the Web. A good starting point for open-source tools is www.opensourcetesting.org/functional.php.

uote:

“The topic of testing is one area in which a good deal of commonality exists between traditional system and client/ server systems.”

Kelley Bourne

WebRef Useful client-sever testing information and resources can be found at www.csst- technologies.com.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 503

Although many different types of tests are conducted at each of these levels of detail, the following testing approaches are commonly encountered for client-server applications:

• Application function tests. The functionality of client applications is tested using the methods discussed earlier in this chapter and in Chapters 19 and 20. In essence, the application is tested in stand-alone fashion in an attempt to uncover errors in its operation.

• Server tests. The coordination and data management functions of the server are tested. Server performance (overall response time and data throughput) is also considered.

• Database tests. The accuracy and integrity of data stored by the server is tested. Transactions posted by client applications are examined to ensure that data are properly stored, updated, and retrieved. Archiving is also tested.

• Transaction tests. A series of tests are created to ensure that each class of transactions is processed according to requirements. Tests focus on the correctness of processing and also on performance issues (e.g., transaction processing times and transaction volume).

• Network communication tests. These tests verify that communication among the nodes of the network occurs correctly and that message passing, transactions, and related network traffic occur without error. Network security tests may also be conducted as part of these tests.

To accomplish these testing approaches, Musa [Mus93] recommends the devel- opment of operational profiles derived from client-server usage scenarios.9 An oper- ational profile indicates how different types of users interoperate with the client-server system. That is, the profiles provide a “pattern of usage” that can be applied when tests are designed and executed. For example, for a particular type of user, what percentage of transactions will be inquiries? updates? orders?

To develop the operational profile, it is necessary to derive a set of scenarios that are similar to use cases (Chapters 5 and 6). Each scenario addresses who, where, what, and why. That is, who the user is, where (in the physical client-server architec- ture) the system interaction occurs, what the transaction is, and why it has occurred. Scenarios can be derived using requirements elicitation techniques (Chapter 5) or through less formal discussions with end users. The result, however, should be the same. Each scenario should provide an indication of the system functions that will be required to service a particular user, the order in which those functions are required, the timing and response that is expected, and the frequency with which each func- tion is used. These data are then combined (for all users) to create the operational profile. In general, testing effort and the number of test cases to be executed are

504 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

What types of tests are

conducted for client-server systems?

?

9 It should be noted that operational profiles can be used in testing for all types of system architec- tures, not just client-server architecture.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 504

allocated to each usage scenario based on frequency of usage and criticality of the functions performed.

18.8.3 Testing Documentation and Help Facilities

The term software testing conjures images of large numbers of test cases prepared to exercise computer programs and the data that they manipulate. Recalling the defi- nition of software presented in Chapter 1, it is important to note that testing must also extend to the third element of the software configuration—documentation.

Errors in documentation can be as devastating to the acceptance of the program as errors in data or source code. Nothing is more frustrating than following a user guide or an online help facility exactly and getting results or behaviors that do not coincide with those predicted by the documentation. It is for this reason that that documentation testing should be a meaningful part of every software test plan.

Documentation testing can be approached in two phases. The first phase, techni- cal review (Chapter 15), examines the document for editorial clarity. The second phase, live test, uses the documentation in conjunction with the actual program.

Surprisingly, a live test for documentation can be approached using techniques that are analogous to many of the black-box testing methods discussed earlier. Graph-based testing can be used to describe the use of the program; equivalence partitioning and boundary value analysis can be used to define various classes of input and associated interactions. MBT can be used to ensure that documented behavior and actual behavior coincide. Program usage is then tracked through the documentation.

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 505

Documentation Testing The following questions should be answered during documentation and/or help facility

testing:

• Does the documentation accurately describe how to accomplish each mode of use?

• Is the description of each interaction sequence accurate?

• Are examples accurate? • Are terminology, menu descriptions, and system

responses consistent with the actual program? • Is it relatively easy to locate guidance within the

documentation? • Can troubleshooting be accomplished easily with the

documentation? • Are the document’s table of contents and index robust,

accurate, and complete?

• Is the design of the document (layout, typefaces, indentation, graphics) conducive to understanding and quick assimilation of information?

• Are all software error messages displayed for the user described in more detail in the document? Are actions to be taken as a consequence of an error message clearly delineated?

• If hypertext links are used, are they accurate and complete?

• If hypertext is used, is the navigation design appropriate for the information required?

The only viable way to answer these questions is to have an independent third party (e.g., selected users) test the documentation in the context of program usage. All discrepancies are noted and areas of document ambiguity or weakness are defined for potential rewrite.

INFO

pre75977_ch18.qxd 11/27/08 6:12 PM Page 505

18.8.4 Testing for Real-Time Systems

The time-dependent, asynchronous nature of many real-time applications adds a new and potentially difficult element to the testing mix—time. Not only does the test- case designer have to consider conventional test cases but also event handling (i.e., interrupt processing), the timing of the data, and the parallelism of the tasks (processes) that handle the data. In many situations, test data provided when a real- time system is in one state will result in proper processing, while the same data pro- vided when the system is in a different state may lead to error.

For example, the real-time software that controls a new photocopier accepts operator interrupts (i.e., the machine operator hits control keys such as RESET or DARKEN) with no error when the machine is making copies (in the “copying” state). These same operator interrupts, if input when the machine is in the “jammed” state, cause a display of the diagnostic code indicating the location of the jam to be lost (an error).

In addition, the intimate relationship that exists between real-time software and its hardware environment can also cause testing problems. Software tests must consider the impact of hardware faults on software processing. Such faults can be extremely difficult to simulate realistically.

Comprehensive test-case design methods for real-time systems continue to evolve. However, an overall four-step strategy can be proposed:

• Task testing. The first step in the testing of real-time software is to test each task independently. That is, conventional tests are designed for each task and executed independently during these tests. Task testing uncovers errors in logic and function but not timing or behavior.

• Behavioral testing. Using system models created with automated tools, it is possible to simulate the behavior of a real-time system and examine its behavior as a consequence of external events. These analysis activities can serve as the basis for the design of test cases that are conducted when the real-time software has been built. Using a technique that is similar to equiva- lence partitioning (Section 18.6.2), events (e.g., interrupts, control signals) are categorized for testing. For example, events for the photocopier might be user interrupts (e.g., reset counter), mechanical interrupts (e.g., paper jammed), system interrupts (e.g., toner low), and failure modes (e.g., roller overheated). Each of these events is tested individually, and the behavior of the executable system is examined to detect errors that occur as a conse- quence of processing associated with these events. The behavior of the system model (developed during the analysis activity) and the executable software can be compared for conformance. Once each class of events has been tested, events are presented to the system in random order and with random frequency. The behavior of the software is examined to detect behavior errors.

506 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

What is an effective

strategy for testing a real-time system?

?

pre75977_ch18.qxd 11/27/08 6:12 PM Page 506

• Intertask testing. Once errors in individual tasks and in system behavior have been isolated, testing shifts to time-related errors. Asynchronous tasks that are known to communicate with one another are tested with different data rates and processing load to determine if intertask synchronization errors will occur. In addition, tasks that communicate via a message queue or data store are tested to uncover errors in the sizing of these data storage areas.

• System testing. Software and hardware are integrated, and a full range of system tests are conducted in an attempt to uncover errors at the software- hardware interface. Most real-time systems process interrupts. Therefore, testing the handling of these Boolean events is essential. Using the state diagram (Chapter 7), the tester develops a list of all possible interrupts and the processing that occurs as a consequence of the interrupts. Tests are then designed to assess the following system characteristics:

• Are interrupt priorities properly assigned and properly handled?

• Is processing for each interrupt handled correctly?

• Does the performance (e.g., processing time) of each interrupt-handling procedure conform to requirements?

• Does a high volume of interrupts arriving at critical times create problems in function or performance?

In addition, global data areas that are used to transfer information as part of interrupt processing should be tested to assess the potential for the generation of side effects.

1 8 . 9 PAT T E R N S F O R S O F T WA R E T E S T I N G The use of patterns as a mechanism for describing solutions to specific design problems was discussed in Chapter 12. But patterns can also be used to propose solutions to other software engineering situations—in this case, software testing. Testing patterns describe common testing problems and solutions that can assist you in dealing with them.

Not only do testing patterns provide you with useful guidance as testing activities commence, they also provide three additional benefits described by Marick [Mar02]:

1. They [patterns] provide a vocabulary for problem-solvers. “Hey, you know, we should

use a Null Object.”

2. They focus attention on the forces behind a problem. That allows [test case] designers

to better understand when and why a solution applies.

3. They encourage iterative thinking. Each solution creates a new context in which new

problems can be solved.

Although these benefits are “soft,” they should not be overlooked. Much of software testing, even during the past decade, has been an ad hoc activity. If testing patterns can help a software team to communicate about testing more effectively;

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 507

WebRef A software testing patterns catalog can be found at www.rbsc .com/pages/ TestPatternList.htm.

Testing patterns can help a software team communicate more effectively about testing and better understand the forces that lead to a specific testing approach.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 507

to understand the motivating forces that lead to a specific approach to testing, and to approach the design of tests as an evolutionary activity in which each iteration results in a more complete suite of test cases, then patterns have accomplished much.

Testing patterns are described in much the same way as design patterns (Chapter 12). Dozens of testing patterns have been proposed in the literature (e.g., [Mar02]). The following three testing patterns (presented in abstract form only) provide representative examples:

Pattern name: PairTesting

Abstract: A process-oriented pattern, PairTesting describes a technique that is anal-

ogous to pair programming (Chapter 3) in which two testers work together to design and

execute a series of tests that can be applied to unit, integration or validation testing

activities.

Pattern name: SeparateTestInterface

Abstract: There is a need to test every class in an object-oriented system, including

“internal classes” (i.e., classes that do not expose any interface outside of the component

that used them). The SeparateTestInterface pattern describes how to create “a test

interface that can be used to describe specific tests on classes that are visible only inter-

nally to a component” [Lan01].

Pattern name: ScenarioTesting

Abstract: Once unit and integration tests have been conducted, there is a need to

determine whether the software will perform in a manner that satisfies users. The

ScenarioTesting pattern describes a technique for exercising the software from the

user’s point of view. A failure at this level indicates that the software has failed to meet a

user visible requirement [Kan01].

A comprehensive discussion of testing patterns is beyond the scope of this book. If you have further interest, see [Bin99] and [Mar02] for additional information on this important topic.

1 8 . 1 0 S U M M A R Y The primary objective for test-case design is to derive a set of tests that have the highest likelihood for uncovering errors in software. To accomplish this objective, two different categories of test-case design techniques are used: white-box testing and black-box testing.

White-box tests focus on the program control structure. Test cases are derived to ensure that all statements in the program have been executed at least once during testing and that all logical conditions have been exercised. Basis path testing, a white-box technique, makes use of program graphs (or graph matrices) to derive the set of linearly independent tests that will ensure statement coverage. Condition and data flow testing further exercise program logic, and loop testing complements other white-box techniques by providing a procedure for exercising loops of varying degrees of complexity.

508 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

WebRef Patterns that describe testing organization, efficiency, strategy, and problem resolution can be found at: www .testing.com/ test-patterns/ patterns/.

pre75977_ch18.qxd 11/27/08 6:12 PM Page 508

Hetzel [Het84] describes white-box testing as “testing in the small.” His implica- tion is that the white-box tests that have been considered in this chapter are typically applied to small program components (e.g., modules or small groups of modules). Black-box testing, on the other hand, broadens your focus and might be called “testing in the large.”

Black-box tests are designed to validate functional requirements without regard to the internal workings of a program. Black-box testing techniques focus on the information domain of the software, deriving test cases by partitioning the input and output domain of a program in a manner that provides thorough test coverage. Equivalence partitioning divides the input domain into classes of data that are likely to exercise a specific software function. Boundary value analysis probes the pro- gram’s ability to handle data at the limits of acceptability. Orthogonal array testing provides an efficient, systematic method for testing systems with small numbers of input parameters. Model-based testing uses elements of the requirements model to test the behavior of an application.

Specialized testing methods encompass a broad array of software capabilities and application areas. Testing for graphical user interfaces, client-server architectures, documentation and help facilities, and real-time systems each require specialized guidelines and techniques.

Experienced software developers often say, “Testing never ends, it just gets trans- ferred from you [the software engineer] to your customer. Every time your customer uses the program, a test is being conducted.” By applying test-case design, you can achieve more complete testing and thereby uncover and correct the highest number of errors before the “customer’s tests” begin.

P R O B L E M S A N D P O I N T S T O P O N D E R 18.1. Myers [Mye79] uses the following program as a self-assessment for your ability to spec- ify adequate testing: A program reads three integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral. Develop a set of test cases that you feel will adequately test this program.

18.2. Design and implement the program (with error handling where appropriate) specified in Problem 18.1. Derive a flow graph for the program and apply basis path testing to develop test cases that will guarantee that all statements in the program have been tested. Execute the cases and show your results.

18.3. Can you think of any additional testing objectives that are not discussed in Section 18.1.1?

18.4. Select a software component that you have designed and implemented recently. Design a set of test cases that will ensure that all statements have been executed using basis path testing.

18.5. Specify, design, and implement a software tool that will compute the cyclomatic com- plexity for the programming language of your choice. Use the graph matrix as the operative data structure in your design.

18.6. Read Beizer [Bei95] or a related Web-based source (e.g., www.laynetworks.com/ Discrete%20Mathematics_1g.htm) and determine how the program you have developed in Problem 18.5 can be extended to accommodate various link weights. Extend your tool to process execution probabilities or link processing times.

C H A P T E R 1 8 T E S T I N G C O N V E N T I O N A L A P P L I C A T I O N S 509

pre75977_ch18.qxd 11/27/08 6:12 PM Page 509

18.7. Design an automated tool that will recognize loops and categorize them as indicated in Section 18.5.3.

18.8. Extend the tool described in Problem 18.7 to generate test cases for each loop category, once encountered. It will be necessary to perform this function interactively with the tester.

18.9. Give at least three examples in which black-box testing might give the impression that “everything’s OK,” while white-box tests might uncover an error. Give at least three examples in which white-box testing might give the impression that “everything’s OK,” while black-box tests might uncover an error.

18.10. Will exhaustive testing (even if it is possible for very small programs) guarantee that the program is 100 percent correct?

18.11. Test a user manual (or help facility) for an application that you use frequently. Find at least one error in the documentation.

F U R T H E R R E A D I N G S A N D I N F O R M AT I O N S O U R C E S Virtually all books dedicated to software testing consider both strategy and tactics. Therefore, further readings noted for Chapter 17 are equally applicable for this chapter. Everett and Raymond (Software Testing, Wiley-IEEE Computer Society Press, 2007), Black (Pragmatic Soft- ware Testing, Wiley, 2007), Spiller and his colleagues (Software Testing Process: Test Management, Rocky Nook, 2007), Perry (Effective Methods for Software Testing, 3d ed., Wiley, 2005), Lewis (Software Testing and Continuous Quality Improvement, 2d ed., Auerbach, 2004), Loveland and his colleagues (Software Testing Techniques, Charles River Media, 2004), Burnstein (Practical Soft- ware Testing, Springer, 2003), Dustin (Effective Software Testing, Addison-Wesley, 2002), Craig and Kaskiel (Systematic Software Testing, Artech House, 2002), Tamres (Introducing Software Testing, Addison-Wesley, 2002), and Whittaker (How to Break Software, Addison-Wesley, 2002) are only a small sampling of many books that discuss testing principles, concepts, strategies, and methods.

A second edition of Myers [Mye79] classic text has been produced by Myers and his col- leagues (The Art of Software Testing, 2d ed., Wiley, 2004) and covers test-case design techniques in considerable detail. Pezze and Young (Software Testing and Analysis, Wiley, 2007), Perry (Effective Methods for Software Testing, 3d ed., Wiley, 2006), Copeland (A Practitioner’s Guide to Software Test Design, Artech, 2003), Hutcheson (Software Testing Fundamentals, Wiley, 2003), Jorgensen (Software Testing: A Craftsman’s Approach, 2d ed., CRC Press, 2002) each provide use- ful presentations of test-case design methods and techniques. Beizer’s [Bei90] classic text pro- vides comprehensive coverage of white-box techniques, introducing a level of mathematical rigor that has often been missing in other treatments of testing. His later book [Bei95] presents a concise treatment of important methods.

Software testing is a resource-intensive activity. It is for this reason that many organizations automate parts of the testing process. Books by Li and Wu (Effective Software Test Automation, Sybex, 2004); Mosely and Posey (Just Enough Software Test Automation, Prentice-Hall, 2002); Dustin, Rashka, and Poston (Automated Software Testing: Introduction, Management, and Perfor- mance, Addison-Wesley, 1999); Graham and her colleagues (Software Test Automation, Addison- Wesley, 1999); and Poston (Automating Specification-Based Software Testing, IEEE Computer Society, 1996) discuss tools, strategies, and methods for automated testing. Nquyen and his col- leagues (Global Software Test Automation, Happy About Press, 2006) present an executive overview of testing automation.

Thomas and his colleagues (Java Testing Patterns, Wiley, 2004) and Binder [Bin99] describe testing patterns that cover testing of methods, classes/clusters, subsystems, reusable compo- nents, frameworks, and systems as well as test automation and specialized database testing.

A wide variety of information sources on test-case design methods is available on the Inter- net. An up-to-date list of World Wide Web references that are relevant to testing techniques can be found at the SEPA website: www.mhhe.com/engcs/compsci/pressman/professional/ olc/ser.htm.

510 P A R T T H R E E Q U A L I T Y M A N A G E M E N T

pre75977_ch18.qxd 11/27/08 6:12 PM Page 510