Software Development Processes: Assignment
Chapter 10: Testing and Quality Assurance
1
Understand quality & basic techniques for software verification and validation.
Analyze basics of software testing and testing techniques.
Discuss the concept of “inspection” process.
Objectives
2
Quality assurance (QA): activities designed to measure and improve quality in a product— and process.
Quality control (QC): activities designed to validate and verify the quality of the product through detecting faults and “fixing” the defects.
Need good techniques, process, tools, and team.
Testing Introduction
similar
3
Two traditional definitions:
Conforms to requirements.
Fit to use.
Verification: checking software conforms to its requirements (did the software evolve from the requirements properly; does the software “work”?)
Is the system correct?
Validation: checking software meets user requirements (fit to use)
Are we building the correct system?
What Is “Quality”?
4
Testing: executing program in a controlled environment and “verifying/validating” output.
Inspections and reviews.
Formal methods (proving software correct).
Static analysis detects “error-prone conditions.”
Some “Error-Detection” Techniques (finding errors)
5
Error: a mistake made by a programmer or software engineer that caused the fault, which in turn may cause a failure
Fault (defect, bug): condition that may cause a failure in the system
Failure (problem): inability of system to perform a function according to its spec due to some fault
Fault or failure/problem severity (based on consequences)
Fault or failure/problem priority (based on importance of developing a fix, which is in turn based on severity)
Faults and Failures
6
Activity performed for:
Evaluating product quality
Improving products by identifying defects and having them fixed prior to software release
Dynamic (running-program) verification of program’s behavior on a finite set of test cases selected from execution domain.
Testing can NOT prove product works 100%—even though we use testing to demonstrate that parts of the software works.
Testing
Not always
done!
7
Who tests?
Programmers
Testers/Req. Analyst
Users
What is tested?
Unit code testing
Functional code testing
Integration/system testing
User interface testing
Testing (cont.)
Why test?
Acceptance (customer)
Conformance (std, laws, etc.)
Configuration (user vs. dev.)
Performance, stress, security, etc.
How (test cases designed)?
Intuition
Specification based (black box)
Code based (white box)
Existing cases (regression)
8
Progression of Testing
Equivalence Class Partitioning
Divide the input into several groups, deemed “equivalent” for purposes of finding errors.
Pick one “representative” for each class used for testing.
Equivalence classes determined by req./design specifications and some intuition
Example: pick “larger” of
two integers and . . .
Lessen duplication.
Complete coverage.
10
Suppose we have n distinct functional requirements.
Suppose further that these n “functional” requirements are such that
r1 U r2 U — U rn = all n requirements and
ri ∩ rj = θ
We can devise a test scenario, ti, for each of the ri functionality to check if ri “works.” Then:
t1 U t2 U — tn = all the test cases to cover the software functionalities.
Note that there may be more than one ti for ri. But picking only one from the set of potential test cases for ri, we form an equivalence class of test cases.
Simple Example of Equivalence Testing
11
Past experiences show that “boundaries” are error prone.
Do equivalence-class partitioning; add test cases for boundaries (at boundary, outside, inside).
Reduced cases: consider boundary as falling between numbers.
If boundary is at 12: normal: 11, 12, 13; reduced: 12, 13 (boundary 12 and 13)
Large number of cases (~3 per boundary).
Good for “ordinal values.”
Boundary Value Analysis (A Black-Box Technique)
12
Boundaries of the Input Values
n
1
1,000,000
age
1
150
The “basic” boundary value testing for a value would include:
At the “minimum” boundary
Immediately above minimum
Between minimum and maximum (nominal)
Immediately below maximum
At the “maximum” boundary
1 <= number of employees, n <= 1,000,000
1 <= employee age <= 150
** Note that we did not include the “outside” of the boundaries here.**
13
A white-box technique
Two tasks
Analyze number of paths in program.
Decide which ones to test.
Decreasing coverage:
Logical paths
Independent paths
Branch coverage
Statement coverage
Path Analysis
<INSERT FIGURE 10.2>
14
A “CASE” Structure
Example with a Loop
A Simple Loop Structure
Linearly Independent Set of Paths
Remember McCabe’s cyclomatic number?
It is the same as linearly independent set of paths.
17
Total # of Paths and Linearly Independent Paths
For functions of several related variables.
To fully test, we need all possible combinations (of equivalence classes).
How to reduce testing:
Coverage analysis.
Assess “important” (e.g., main functionalities) cases.
Test all pairs of relations (but not all combinations).
Combinations of Conditions
19
Unit Testing: Test each individual unit.
Usually done by the programmer.
Test each unit as it is developed (small chunks).
Keep test cases/results around (use Junit or xxxUnit).
Allows for regression testing.
Facilitates refactoring.
Tests become documentation !!
Unit Testing
20
Write unit test cases BEFORE the code!
Test cases “are”/“become” requirements.
Forces development in small steps.
Steps:
Write test case and code.
Verify (it fails or runs).
Modify code so it succeeds.
Rerun test case, previous tests.
Refactor until (success and satisfaction).
Test-Driven Development
21
Simple answer: stop when
All planned test cases are executed.
All problems that are found are fixed.
Other techniques:
Stop when you are not finding any more errors.
Defect seeding — test until all (or % of ) the seeded bugs are found.
NOT when you ran out of time — poor planning!
When to Stop Testing?
22
Seed the program (component):
Generate and scatter with “x” number of bugs.
Do not tell the testers.
Set a % (e.g., 95%) of seed bugs found as stopping criteria.
Suppose “y” number of the “x” seed bugs are found:
If (y/x) > (stopping percentage), stop testing.
If (y/x) ≤ (stopping percentage), keep on testing.
Get a feel of how many bugs may still remain:
Suppose you discovered “u” non-seeded bugs through testing.
Set y/x = u/v; v = (u * x)/y.
Then there is most likely (v − u) bugs still left in the software.
Defect Seeding
23
Problem Find Rate
Decreasing Problem Find Rate
Review: any process involving human testers reading and understanding a document and then analyzing it with the purpose of detecting errors
Walkthrough: author explaining document to team of people
Software inspection: detailed reviews of work in progress, following Fagan’s method
Inspections and Reviews
25
Steps:
Planning
Overview
Preparation
Inspection
Rework
Follow-up
Focused on finding defects
Output: list of defects
Team of:
3−6 people
Author included
People working on related efforts
Moderator, reader, scribe
Software Inspections
26
Inspections
Partially cost-effective.
Can be applied to intermediate artifacts.
Catches defects early.
Helps disseminate knowledge about project and best practices.
Testing
Finds errors cheaper, but correcting them is expensive.
Can only be applied to code.
Catches defects late (after implementation).
Necessary to gauge quality.
Inspections vs. Testing
27
Mathematical techniques used to prove that a program works.
Used for requirements/design/algorithm specification.
Prove that implementation conforms to spec.
Pre and post conditions
Problems:
Require math training.
Not applicable to all programs.
Only verification, not validation.
Not applicable to all aspects of program (e.g., UI or maintainability).
Formal Methods
28
Examination of static structures of design/code for detecting error-prone conditions (cohesion — coupling).
Automatic program tools are more useful.
Can be applied to:
Intermediate documents (but in formal model)
Source code
Executable files
Output needs to be checked by programmer.
Static Analysis
29