CIS 518 Case Study 2

Chapter 9

Testing the System

Shari L. Pfleeger

Joann M. Atlee

4th Edition

4th Edition

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

Contents

9.1 Principles of system testing

9.2 Function testing

9.3 Performance testing

9.4 Reliability, availability, and maintainability

9.5 Acceptance testing

9.6 Installation testing

9.7 Automated system testing

9.8 Test documentation

9.9 Testing safety-critical systems

9.10 Information systems example

9.11 Real-time example

9.12 What this chapter means for you

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

Chapter 9 Objectives

• Function testing
• Performance testing
• Acceptance testing
• Software reliability, availability, and maintainability
• Installation testing
• Test documentation
• Testing safety-critical systems

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Source of Software Faults During Development

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing System Testing Process

• Function testing: does the integrated system perform as promised by the requirements specification?
• Performance testing: are the non-functional requirements met?
• Acceptance testing: is the system what the customer expects?
• Installation testing: does the system run at the customer site(s)?

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
System Testing Process (continued)

• Pictorial representation of steps in testing process

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Techniques Used in System Testing

• Build or integration plan
• Regression testing
• Configuration management
• versions and releases
• production system vs. development system
• deltas, separate files and conditional compilation
• change control

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Build or Integration Plan

• Define the subsystems (spins) to be tested
• Describe how, where, when, and by whom the tests will be conducted

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Example of Build Plan for Telecommunication System

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Example Number of Spins for Star Network

Spin 0: test the central computer’s general functions

Spin 1: test the central computer’s message-translation function

Spin 2: test the central computer’s message-assimilation function

Spin 3: test each outlying computer in the stand alone mode

Spin 4: test the outlying computer’s message-sending function

Spin 5: test the central computer’s message-receiving function

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Regression Testing

• Identifies new faults that may have been introduced as current one are being corrected
• Verifies a new version or release still performs the same functions in the same manner as an older version or release

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Regression Testing Steps

• Inserting the new code
• Testing functions known to be affected by the new code
• Testing essential function of m to verify that they still work properly
• Continuing function testing m + 1

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Configuration Management

• Versions and releases
• Production system vs. development system
• Deltas, separate files and conditional compilation
• Change control

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.1 Principles of System Testing
Test Team

• Professional testers: organize and run the tests
• Analysts: who created requirements
• System designers: understand the proposed solution
• Configuration management specialists: to help control fixes
• Users: to evaluate issues that arise

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.2 Function Testing
Purpose and Roles

• Compares the system’s actual performance with its requirements
• Develops test cases based on the requirements document

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.2 Function Testing
Cause-and-Effect Graph

• A Boolean graph reflecting logical relationships between inputs (causes), and the outputs (effects) or transformations (effects)

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.2 Function Testing
Notation for Cause-and-Effect Graph

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.2 Function Testing
Cause-and-Effect Graphs Example

• INPUT: The syntax of the function is LEVEL(A,B) where A is the height in meters of the water behind the dam, and B is the number of centimeters of rain in the last 24-hour period
• PROCESSING: The function calculates whether the water level is within a safe range, is too high, or is too low
• OUTPUT: The screen shows one of the following messages

1. “LEVEL = SAFE” when the result is safe or low

2. “LEVEL = HIGH” when the result is high

3. “INVALID SYNTAX”

depending on the result of the calculation

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.2 Function Testing
Cause-and-Effect Graphs Example (Continued)

• Causes

The first five characters of the command “LEVEL”

The command contains exactly two parameters separated by a comma and enclosed in parentheses

The parameters A and B are real numbers such that the water level is calculated to be LOW

The parameters A and B are real numbers such that the water level is calculated to be SAFE

The parameters A and B are real numbers such that the water level is calculated to be HIGH

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.2 Function Testing
Cause-and-Effect Graphs Example (Continued)

• Effects

1. The message “LEVEL = SAFE” is displayed on the screen

2. The message “LEVEL = HIGH” is displayed on the screen

The message “INVALID SYNTAX” is printed out

Intermediate nodes

1. The command is syntactically valid

2. The operands are syntactically valid

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.2 Function Testing
Cause-and-Effect Graphs of LEVEL Function Example

Exactly one of a set of conditions can be invoked

At most one of a set of conditions can be invoked

At least one of a set of condition can be invoked

One effects masks the observance of another effect

Invocation of one effect requires the invocation of another

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.2 Function Testing
Decision Table for Cause-and-Effect Graph of LEVEL Function

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.3 Performance Tests
Purpose and Roles

• Used to examine
• the calculation
• the speed of response
• the accuracy of the result
• the accessibility of the data
• Designed and administrated by the test team

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.3 Performance Tests
Types of Performance Tests

Stress tests

Volume tests

Configuration tests

Compatibility tests

Regression tests

Security tests

Timing tests

Environmental tests

Quality tests

Recovery tests

Maintenance tests

Documentation tests

Human factors (usability) tests

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Definition

• Software reliability: operating without failure under given condition for a given time interval
• Software availability: operating successfully according to specification at a given point in time
• Software maintainability: for a given condition of use, a maintenance activity can be carried out within stated time interval, procedures and resources

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Different Level of Failure Severity

• Catastrophic: causes death or system loss
• Critical: causes severe injury or major system damage
• Marginal: causes minor injury or minor system damage
• Minor: causes no injury or system damage

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Failure Data

• Table of the execution time (in seconds) between successive failures of a command-and-control system

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Failure Data (Continued)

• Graph of failure data from previous table

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Uncertainty Inherent from Failure Data

• Type-1 uncertainty: how the system will be used
• Type-2 uncertainty: lack of knowledge about the effect of fault removal

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Measuring Reliability, Availability, and Maintainability

• Mean time to failure (MTTF)
• Mean time to repair (MTTR)
• Mean time between failures (MTBF)

MTBF = MTTF + MTTR

• Reliability R = MTTF/(1+MTTF)
• Availability A = MTBF (1+MTBF)
• Maintainability M = 1/(1+MTTR)

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Reliability Stability and Growth

• Probability density function f or time t, f (t): when the software is likely to fail
• Distribution function: the probability of failure
• F(t) = ∫ f (t) dt
• Reliability Function: the probability that the software will function properly until time t
• R(t) = 1- F(t)

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Uniformity Density Function

• Uniform in the interval from t=0..86,400 because the function takes the same value in that interval

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Sidebar 9.4 Difference Between Hardware and Software Reliability

• Complex hardware fails when a component breaks and no longer functions as specified
• Software faults can exist in a product for long time, activated only when certain conditions exist that transform the fault into a failure

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Reliability Prediction

• Predicting next failure times from past history

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Elements of a Prediction System

• A prediction model: gives a complete probability specification of the stochastic process
• An inference procedure: for unknown parameters of the model based on values of t₁, t₂, …, ti-1
• A prediction procedure: combines the model and inference procedure to make predictions about future failure behavior

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Reliability Model

• The Jelinski-Moranda model: assumes
• no type-2 uncertainty
• corrections are perfect
• fixing any fault contributes equally to improving the reliability
• The Littlewood model
• treats each corrected fault’s contribution to reliability as independent variable
• uses two source of uncertainty

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Successive Failure Times for Jelinski-Moranda

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.5 Acceptance Tests
Purpose and Roles

• Enable the customers and users to determine if the built system meets their needs and expectations
• Written, conducted and evaluated by the customers

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.5 Acceptance Tests
Types of Acceptance Tests

• Pilot test: install on experimental basis
• Alpha test: in-house test
• Beta test: customer pilot
• Parallel testing: new system operates in parallel with old system

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.4 Reliability, Availability, and Maintainability
Result of Acceptance Tests

• List of requirements
• are not satisfied
• must be deleted
• must be revised
• must be added

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.6 Installation Testing

• Before the testing
• Configure the system
• Attach proper number and kind of devices
• Establish communication with other system
• The testing
• Regression tests: to verify that the system has been installed properly and works

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.7 Automated System Testing
Simulator

• Presents to a system all the characteristics of a device or system without actually having the device or system available
• Looks like other systems with which the test system must interface
• Provides the necessary information for testing without duplication the entire other system

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation

• Test plan: describes system and plan for exercising all functions and characteristics
• Test specification and evaluation: details each test and defines criteria for evaluating each feature
• Test description: test data and procedures for each test
• Test analysis report: results of each test

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Documents Produced During Testing

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Test Plan

• The plan begins by stating its objectives, which should
• guide the management of testing
• guide the technical effort required during testing
• establish test planning and scheduling
• explain the nature and extent of each test
• explain how the test will completely evaluate system function and performance
• document test input, specific test procedures, and expected outcomes

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Parts of a Test Plan

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Testing Documentation
Test-Requirement Correspondence Chart

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Sidebar 9.8 Measuring Test Effectiveness and Efficiency

• Test effectiveness can be measured by dividing the number of faults found in a given test by the total number of faults found
• Test efficiency is computed by dividing the number of faults found in testing by the effort needed to perform testing

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Test Description

• Including
• the means of control
• the data
• the procedures

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Test Description Example

INPUT DATA:

Input data are to be provided by the LIST program. The program generates randomly a list of N words of alphanumeric characters; each word is of length M. The program is invoked by calling

RUN LIST(N,M)

in your test driver. The output is placed in global data area LISTBUF. The test datasets to be used for this test are as follows:

Case 1: Use LIST with N=5, M=5

Case 2: Use LIST with N=10, M=5

Case 3: Use LIST with N=15, M=5

Case 4: Use LIST with N=50, M=10

Case 5: Use LIST with N=100, M=10

Case 6: Use LIST with N=150, M=10

INPUT COMMANDS:

The SORT routine is invoked by using the command

RUN SORT (INBUF,OUTBUF) or

RUN SORT (INBUF)

OUTPUT DATA:

If two parameters are used, the sorted list is placed in OUTBUF. Otherwise, it is placed in INBUF.

SYSTEM MESSAGES:

During the sorting process, the following message is displayed:

“Sorting ... please wait ...”

Upon completion, SORT displays the following message on the screen:

“Sorting completed”

To halt or terminate the test before the completion message is displayed, press CONTROL-C on the keyboard.

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Test Script for Testing The “change field”

Step N: Press function key 4: Access data file.

Step N+1: Screen will ask for the name of the date file.

Type ‘sys:test.txt’

Step N+2: Menu will appear, reading

* delete file

* modify file

* rename file

Place cursor next to ‘modify file’ and press RETURN key.

Step N+3: Screen will ask for record number. Type ‘4017’.

Step N+4: Screen will fill with data fields for record 4017:

Record number: 4017 X: 0042 Y: 0036

Soil type: clay Percolation: 4 mtrs/hr

Vegetation: kudzu Canopy height: 25 mtrs

Water table: 12 mtrs Construct: outhouse

Maintenance code: 3T/4F/9R

Step N+5: Press function key 9: modify

Step N+6: Entries on screen will be highlighted. Move cursor to VEGETATION field. Type ‘grass’ over ‘kudzu’ and press RETURN key.

Step N+7: Entries on screen will no longer be highlighted.

VEGETATION field should now read ‘grass’.

Step N+8: Press function key 16: Return to previous screen.

Step N+9: Menu will appear, reading

* delete file

* modify file

* rename file

To verify that the modification has been recorded,place cursor next to ‘modify file’ and press RETURN key.

Step N+10: Screen will ask for record number. Type ‘4017’.

Step N+11: Screen will fill with data fields for record 4017:

Record number: 4017 X: 0042 Y: 0036

Soil type: clay Percolation: 4 mtrs/hr

Vegetation: grass Canopy height: 25 mtrs

Water table: 12 mtrs Construct: outhouse

Maintenance code: 3T/4F/9R

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Test Analysis Report

• Documents the result of test
• Provides information needed to duplicate the failure and to locate and fix the source of the problem
• Provides information necessary to determine if the project is complete
• Establish confidence in the system’s performance

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.8 Test Documentation
Problem Report Forms

• Location: Where did the problem occur?
• Timing: When did it occur?
• Symptom: What was observed?
• End result: What were the consequences?
• Mechanism: How did it occur?
• Cause: Why did it occur?
• Severity: How much was the user or business affected?
• Cost: How much did it cost?

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.10 Information Systems Example
The Piccadilly System

• Many variables, many different test cases to consider
• An automated testing tool may be useful

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.10 Information Systems Example
Things to Consider in Selecting a Test Tool

• Capability
• Reliability
• Capacity
• Learnability
• Operability
• Performance
• Compatibility
• Nonintrusiveness

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.10 Information Systems Example
Sidebar 9.13 Why Six-Sigma Efforts Do Not Apply to Software

• A six-sigma quality constraint says that in a billion parts, we can expect only 3.4 to be outside the acceptable range
• It is not apply to software because
• People are variable, the software process inherently contains a large degree of uncontrollable variation
• Software either conforms or it does not, there are no degree of conformance
• Software is not the result of a mass-production process

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.11 Real-Time Example
Ariane-5 Failure

• Simulation might help preventing the failure
• Could have generated signals related to predicted flight parameters while turntable provided angular movement

Pfleeger and Atlee, Software Engineering: Theory and Practice

Chapter 9.*

9.12 What This Chapter Means for You

• Should anticipate testing from the very beginning of the system life cycle
• Should think about system functions during requirement analysis
• Should use fault-tree analysis, failure modes and effect analysis during design
• Should build safety case during design and code reviews
• Should consider all possible test cases during testing