RESEARCH SYNTHESIS

gnsv.srinivas

Professional_Scrum_with_Team_Foundation_Server_201..._----_Chapter_7_Tracking_Quality.pdf

Home >Business & Finance homework help >Operations Management homework help >RESEARCH SYNTHESIS

Tracking Quality

WHAT ’ S IN THIS CHAPTER?

Understanding what to measure.

Tracking and resolving bugs and tasks.

Creating and running test cases.

Using Microsoft Test Manager.

Quality is generally easy to understand and observe. It ’ s extremely subjective, but people can discern good quality from poor quality. A car that reliably starts and drives quietly down the road is good quality; one that requires a screw driver to start and sputters as you drive is clearly bad. Clean, fresh air is good quality; smog is bad.

Just as it ’ s easy to understand quality, it ’ s often easy to measure it. Measuring the loudness of engine noise is straightforward, using decibels as the metric. Measuring air pollution is also pretty simple, using parts per million as the metric. Both of these measurements involve sampling the system as its running, whether it ’ s a combustion engine or the wind, and then counting the results. Samples can be taken under varying conditions, such as sampling engine noise at varying speeds and sampling air quality at different locations.

Once samples are measured, they can then be classifi ed into something people care about. In the case of engine noise, quiet, average, and loud are simple categories. With air quality, good, average, and bad are well understood.

Finally, after quality is understood, measured, and classifi ed, you can take action. You can choose to buy a quiet car or ignore the noise and go with a fast car. Knowing that the air quality is poor, you can choose to exercise indoors rather than go for a run outside.

➤

c07.indd 177c07.indd 177 3/24/11 4:26:54 PM3/24/11 4:26:54 PM

Resnick, Steve, et al. Professional Scrum with Team Foundation Server 2010, John Wiley & Sons, Incorporated, 2011. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/harrisburg-ebooks/detail.action?docID=706927. Created from harrisburg-ebooks on 2020-12-01 07:28:16.

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

178 ❘ CHAPTER 7 TRACKING QUALITY

In this chapter, you will learn how to measure and track quality, which is an important part of Scrum. You ’ ll learn how to do this with test cases, bugs, and tasks in Visual Studio, Excel, and Team Foundation Server (TFS).

KNOWING WHAT TO MEASURE

Great software always has two common attributes: It ’ s useful and predictable. Defi ning and running test cases for software ensures that the product is useful for its intended purpose; tracking and eliminating bugs during the construction phase ensures that it ’ s predictable.

There are a variety of ways to measure software quality. You can measure data accuracy, system performance, reliability, or usability. Regardless of the measure, you will have a specifi c expectation of the result. Most software will have hundreds of such expectations that are ultimately codifi ed into test cases. In aggregate, if the software meets the expectation, then it passes that quality metric. If not, it fails.

If software does not meet an expectation, you have a quality problem. It may be a very small problem, or it may be quite large — but either way, the problem exists. One way to measure software quality is to quantify the expectations that are not met. You can do this by counting the failed test cases. Another way is to quantify the technical items that must be fi xed in order to meet expectations. You can do this by counting the number of open items, or bugs . If you’re counting bugs, it’s helpful to also count the estimated effort necessary to fi x them. This is an indirect measure of quality, but it’s crucial for planning.

It ’ s important for a team to estimate the effort required to fi x a bug, just as it does with product backlog items (PBIs). Bugs ultimately get scheduled like PBIs in later sprints. During sprint planning, the team builds the sprint backlog by picking PBIs and bugs from the product backlog that add up to its velocity. If a team doesn ’ t estimate bug effort, it cannot accurately commit to fi xing the bugs in a sprint.

The combination of usefulness (passing test cases) and predictability (no bugs) is what defi nes quality. In other words, if the system does what the customer expects it to do and it does so predictably, then quality is high.

TRACKING AND RESOLVING BUGS AND TASKS

Fewer bugs equates to greater quality. Having 0 open bugs is ideal, but the bug count alone is an insuffi cient measure for managing quality. How do 10 open low - severity bugs compare to 3 open high - severity bugs? How do bugs in non - essential components compare with bugs in critical parts of the system?

The bug count has a subjective component and an objective component. Labeling a bug as low severity versus high severity is a subjective decision. On the other hand, counting the number of open bugs in each category is objective.

c07.indd 178c07.indd 178 3/24/11 4:26:56 PM3/24/11 4:26:56 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

Of course, bugs don ’ t fi x themselves; people fi x them. The fundamental way to assign work in TFS is to use tasks. For the bugs that will be fi xed within the current sprint, you need to track tasks so people can allocate their time.

The Workfl ow for Resolving Bugs

Whether a bug is addressed during the sprint in which it is found or placed on the backlog for a future sprint, the team should have a workfl ow for tracking and resolving each bug. The Scrum 1.0 template for TFS implements a workfl ow for this.

When to Track and Resolve a Bug

You should track each bug from the moment you discover it until it is ultimately fi xed. Certain bugs can be resolved within the current sprint, while others can be pushed to the product backlog and addressed in a later sprint. The product owner needs to make this determination, based on the product features that the bug affects.

If a bug relates to a PBI being addressed in the current sprint, then the bug should be fi xed in the current sprint. You track a bug in three steps:

1. Enter a new bug as a new work item in the current sprint.

2. Create a new task or set of tasks to track the work required to fi x the bug.

3. Link the task to the bug.

On the other hand, if a discovered bug relates to a feature that is not being worked on in the current sprint, you should add the new bug to the product backlog, but you should not assign any tasks to it. This way, the bug will not affect the current sprint, and it will not get lost.

For instance, assume that the goal of your current sprint is to implement an order entry screen in a web application. When testing the screen, you discover that date validation is not working. After 30 minutes of analysis, you determine that some business logic needs to change. You estimate that it will take four hours to fi x and test the change. For this example, you would create a bug in the current sprint and create a task, linked to the bug, with an effort of 4.

If, however, your 30 - minute analysis reveals that the date validation error is due to the user ’ s time zone crossing the International Date Line from the server, and the system design doesn ’ t address that scenario, you have a larger problem that may ripple through the system. In this case, it would be better to treat the bug as a new PBI. You would create the bug item but not assign it to a sprint and not create tasks.

Figure 7 - 1 shows the decision process for entering new bugs and their associated tasks into TFS and determining whether to track them on the product backlog or in the current sprint.

Defect

discovered

Create bug

in product

backlog

No Yes

Impacts

PBIs in

current sprint?

Create bug

current sprint

Create tasks in

current sprint

FIGURE 7 - 1: Determining when to track

new bugs.

Tracking and Resolving Bugs and Tasks ❘ 179

c07.indd 179c07.indd 179 3/24/11 4:27:12 PM3/24/11 4:27:12 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

180 ❘ CHAPTER 7 TRACKING QUALITY

How to Track Bug State from New to Done

It is important to track progress from when a bug is fi rst detected to when it is ultimately resolved. You capture this information in the State fi eld of a bug item. Figure 7 - 2 shows the transitions built into the Scrum template.

New

Approved Removed

Committed Work stopped

Approved

New bug

Removed

Reconsidered

Done

Committed

Finished More work found

FIGURE 7 - 2: Bug state diagram.

When a bug is fi rst detected, a new bug work item is created and assigned to a product owner. When the product owner reviews the bug, he or she changes its state to approved or removed. If the product owner approves the bug, he or she should set the Backlog Priority, Severity, and Area fi elds and should ensure that the Steps to Reproduce fi eld is adequately detailed. The product owner can set the iteration to the project, putting it on the product backlog, or to the current sprint, placing it on the sprint backlog. If the product owner sets the state to removed, the bug will not be considered on any future sprint.

Before the product owner can put a new bug into the sprint backlog, the team must estimate the effort and then commit to fi xing that bug. The team ’ s commitment is necessary in order to mark the PBI as done before the end of the sprint. To put the bug on the sprint backlog, the product owner does the following:

Sets the State fi eld to Committed

Updates the Iteration fi eld to the current sprint

If the product owner wants to put the new bug in the product backlog rather than in the sprint backlog, the bug remains assigned to the product owner. It remains in the approved state but is not

➤

c07.indd 180c07.indd 180 3/24/11 4:27:12 PM3/24/11 4:27:12 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

committed, and the iteration is the project. The bug will be considered in future sprints, but it will not be fi xed within the current sprint.

Once a bug is committed, the team creates a task or tasks to fi x the bug and assigns the task(s) to the team member(s) who will be fi xing it. Each of these new tasks is linked to the bug. Each task can be assigned to a different team member, and each team member is responsible for updating the Remaining Work fi eld.

Entering Bugs

You can enter a bug in Visual Studio, Excel, or SharePoint. Whichever tool you chose, the information is stored in TFS so it is accessible to other tools and reports.

To enter a bug in Visual Studio, you select Team ➪ New Bug. Then you fi ll in the following fi elds:

Title

Iteration

Assigned To

State

Backlog Priority

Effort

Severity

Area

These fi elds are not all required in Visual Studio, but throughout the fl ow and life cycle of a bug, they are all used. The following sections describe them in more detail, and Figure 7 - 3 shows the Visual Studio screen where you enter bugs. You reach this screen by selecting Team ➪ New Bug.

➤

FIGURE 7 - 3: Bug entry in Visual Studio.

Tracking and Resolving Bugs and Tasks ❘ 181

c07.indd 181c07.indd 181 3/24/11 4:27:12 PM3/24/11 4:27:12 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

182 ❘ CHAPTER 7 TRACKING QUALITY

Title

In the Title fi eld you enter a short description of the bug. The title you enter here will appear on lists and reports in Excel, Visual Studio, and SharePoint. Shorter titles are better, as they are more manageable in Excel, where you often print or view many columns. The title should convey distinguishing features of the bug, such as “ crashes on low memory systems ” or “ can enter % over 100 in discount fi eld. ”

Iteration

In the Iteration fi eld you enter the sprint in which the team committed to fi xing the bug. Bugs are very similar to PBIs in that they go onto the product backlog when they are waiting to be scheduled. TFS defaults to the project, indicating that it ’ s on the product backlog. A bug is moved to the sprint backlog during sprint planning when the team sets this fi eld to the current sprint.

It ’ s common for a team to fi x bugs as soon as it fi nds them, within the current sprint. In that case, you set this fi eld to the current sprint and immediately create a task for someone to fi x the big.

Assigned To

In the Assigned To fi eld you select the name of the person responsible for prioritizing the bug within the backlog — typically the product owner. It ’ s important to note that bugs are not assigned to team members; tasks are. Each bug should spawn a task, and the task is then assigned to a team member.

State

In the State fi eld you track the current state of the bug. A bug can be in one of fi ve states, starting with new and ending with either removed or done:

New — The bug is entered into TFS. As a new bug, it hasn ’ t yet been reviewed by the team or product owner, nor is it scheduled to be fi xed.

Approved — The bug has been approved by the product owner and team. Collectively, the team has estimated the effort (for example, in story points), classifi ed it into an area path, and assigned a severity.

Removed — The bug is no longer tracked. This is useful when you ’ re pruning the bug list to remove duplicates, to consolidate similar bugs, or to simply remove the bugs that were reported erroneously and will not be fi xed.

Committed — The bug has been assigned to a sprint, and the team has committed to fi xing it.

Done — The bug is fi xed, and the product owner has tested and verifi ed it.

➤

BUG AND PBI STATES

You will notice that the state diagram and transitions defi ned for a bug are identical to those defi ned for a PBI. This is because both PBIs and bugs are work items in TFS, and they are treated the same way in Scrum for tracking purposes.

c07.indd 182c07.indd 182 3/24/11 4:27:13 PM3/24/11 4:27:13 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

Backlog Priority

In the Backlog Priority fi eld you enter the relative priority of fi xing the bug, compared to items in the backlog. Because the backlog is typically sorted by priority in ascending order, a bug or PBI with a lower priority value appears earlier in the list. Bugs default to priority1,000, which places them at the bottom of the list.

Eff ort

In the Effort fi eld you enter the relative effort to fi x the bug. As with a PBI, this is not expressed in hours or days, but it is a relative value of effort. Sometimes this is estimated as in story points (complexity) or ideal days (which don ’ t exist in nature). It ’ s a best practice to use numbers in the Fibonacci sequence to estimate PBIs, where you have greater resolution with small estimates than you do with large estimates. This works for estimating PBIs because their complexity varies greatly.

Severity

In the Severity fi eld you enter the severity of the bug to the product, from critical to low priority. The team uses this setting to prioritize work and schedule the bug for a sprint.

Area

In the Area fi eld you enter the component of the product in which the bug exists. This fi eld is present on all work items in TFS, but it is especially important for bugs. It ’ s common for the product owner to view the current bug list by area, since the area indicates the readiness or quality of a specifi c component. It ’ s common for the team to view the current bug list this way as well, since different people typically work on different areas of the product so they can quickly assess how their components are doing.

Viewing the Bug List

Once you have begun capturing bugs using the basic attributes just described, you can easily classify the bugs. By using a simple query in TFS, you can pull all the bugs into Excel to sort, fi lter, or pivot them for useful aggregations. The following classifi cations are common:

Open bugs by Area setting

Open bugs by Severity setting

Open bugs by Assigned To setting

➤

Unlike with PBIs, the effort associated with a bug is generally small. It may be estimated at 1 or 0.5.

Tracking and Resolving Bugs and Tasks ❘ 183

c07.indd 183c07.indd 183 3/24/11 4:27:14 PM3/24/11 4:27:14 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

184 ❘ CHAPTER 7 TRACKING QUALITY

You can create a list and chart in Excel of the open bugs by area, indicating how many are approved and how many are committed. To do so, you open Visual Studio and follow these steps:

1. Create a query for listing only the bugs in the current release. To do so, open the sprint backlog query under the Current Sprint folder in the Team Explorer. (You ’ re starting with the sprint backlog query because it has the fi lter defi ned almost the way you want it for this list.)

2. Click Edit Query so you can change the query options.

3. Change the Type of Query setting from Tree of Items to Flat List (Default).

4. Next to Work Item Type, set Operator to � and set Value to Bug.

5. Click the Add Columns button and add Area Path and Severity.

6. Select File ➪ Save Sprint Backlog [editor] As to save the query you modifi ed. Name the query Bugs in Current Sprint either in the My Queries folder or the Team Queries folder.

7. Click Run to see the bugs in the current sprint. If you don ’ t have enough bugs entered, you can change the Iteration fi eld to the project rather than the sprint. The resulting list should look similar to the one shown in Figure 7 - 4.

FIGURE 7 - 4: Bugs in the current sprint in Visual Studio.

c07.indd 184c07.indd 184 3/24/11 4:27:18 PM3/24/11 4:27:18 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

9. In Excel, select all the data, including column headers. For example, because there are 13 bugs listed in Figure 7 - 5, in this example, you would select cells A2 through H14.

10. On the Insert tab in Excel, click PivotChart under the PivotTable button. Excel prompts you for where to place the PivotChart.

11. Choose to place the PivotChart in a new worksheet.

12. From the Design tab in Excel, click Change Chart Type and choose Stacked Column.

13. Drag Area Path from the Choose Fields to Add to Report section to the Axis Fields (Categories) section.

14. Drag State from the Choose Fields to Add to Report section to the � Values section. It will default to Count of State.

15. Drag State, for a second time, from the Choose Fields to Add to Report section to the Legend Fields (Series) section.

Excel should now look as shown in Figure 7 - 6, showing a chart with the number of approved and committed bugs in the current sprint.

FIGURE 7 - 5: The Bugs in Current Sprint query in Excel.

Tracking and Resolving Bugs and Tasks ❘ 185

8. Select Open in Microsoft Offi ce ➪ Open the Query in Microsoft Excel 2007 or Later. Excel opens, as shown in Figure 7 - 5.

c07.indd 185c07.indd 185 3/24/11 4:27:19 PM3/24/11 4:27:19 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

186 ❘ CHAPTER 7 TRACKING QUALITY

Now that you ’ ve seen the workfl ow for bugs and how to enter and query them, you ’ re ready to learn to do the same for tasks.

The Workfl ow for Tasks

Tasks are work items that capture the work necessary to fi x a bug or to implement a PBI. Individually, they represent the work associated with one component; collectively, they represent the total amount of work in a sprint.

As shown in Figure 7-7, tasks go through a simple workfl ow. They begin in the new state, move through in progress state, and conclude in the done state.

Entering Tasks

You can enter a task in Visual Studio, Excel, or SharePoint. Whichever tool you chose, the information is stored in TFS and is accessible to other tools and reports. To enter a task in Visual Studio, you select Team ➪ New Task. Figure 7 - 8 shows the task entry screen that appears.

FIGURE 7 - 6: An Excel PivotChart showing the bugs in the current sprint.

New

Work

started

Work

stopped

New task

Removed

Done

In progress

Finished More work

found

Removed

FIGURE 7 - 7: Task workfl ow.

c07.indd 186c07.indd 186 3/24/11 4:27:20 PM3/24/11 4:27:20 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

Tasks store similar elements to bugs and PBIs for categorization and prioritization, and they also have a few fi elds specifi c to tracking work:

Remaining Work

Activity

Blocked

Remaining Work

In the Remaining Work fi eld you enter the amount of work remaining to complete the task. When estimating remaining work, you should think in terms of time — that is, hours. This is different from PBIs and bugs, where you estimate in story points or some other arbitrary measure.

➤

FIGURE 7 - 8: The task entry screen.

The minimum amount of time you should allocate for a task is 2 hours. Nothing ever takes less than that. The maximum amount of time for a task should be 40 hours, or 1 week. Greater than that, and you probably don ’ t know how long it will take, so it ’ s better to break the task into subtasks.

Activity

In the Activity fi eld you enter the type of work required for the task. The Scrum template allows the following fi eld values:

Design

Development

➤

Tasks are hours. Hours are time. Time is money.

Tracking and Resolving Bugs and Tasks ❘ 187

c07.indd 187c07.indd 187 3/24/11 4:27:21 PM3/24/11 4:27:21 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

188 ❘ CHAPTER 7 TRACKING QUALITY

Deployment

Documentation

Requirements

Testing

➤

Blocked

You can leave the Blocked fi eld blank or enter Yes to indicate whether something is blocking the task from being complete.

The Unfi nished Work Query

The unfi nished work query, shown in Figure 7 - 9, lists the PBIs, bugs, and tasks in the current sprint. Together, these work items comprise the work remaining in the sprint. The unfi nished work query is very useful when you are assessing how much work is remaining and how much of that work deals with quality issues.

You run this query by selecting Unfi nished Work under Team Queries in the Team Explorer.

The Activity fi eld is helpful when you ’ re planning who will be working on what and want to sort the list by activity. It ’ s also helpful for planning. For instance, if you have 180 hours of tasks for deployment activities but only one person who does deployment working for the remaining week of the sprint, you know you need to adjust something.

FIGURE 7 - 9: The unfi nished work query.

c07.indd 188c07.indd 188 3/24/11 4:27:26 PM3/24/11 4:27:26 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

TEST CASES

Setting quality expectations is crucial for ensuring product acceptance. Expectations can be defi ned in terms of performance, security, usability, or other attributes. Regardless of how quality expectations are defi ned, they need to be measured. And for anything that is going to be measured, the team needs to defi ne how, when, and by whom the testing will occur.

Quality is associated with PBIs. For each PBI, you should defi ne its acceptance criteria and create one or more test cases to verify that the PBI meets those criteria. Typically, a product manager defi nes the acceptance criteria as he or she creates the PBI. Then, when a PBI is allocated to a sprint, the product manager or team member creates test cases.

Acceptance Criteria in a PBI

Acceptance criteria should be at a similar level to the PBI description. More detail is helpful, but with Scrum, the subtle communication of acceptance criteria occurs between the product owner and the team member implementing the feature. Greater detail should be provided in the specifi c test cases that are used to verify the PBI.

You should enter acceptance criteria in each PBI. These criteria should be specifi c about how the feature will be tested. If you have performance metrics that must be met or specifi c data validation requirements, it is helpful to include that information here.

Figure 7 - 10 shows an example of acceptance criteria in a PBI.

FIGURE 7 - 10: Acceptance criteria in a PBI.

Test Cases ❘ 189

c07.indd 189c07.indd 189 3/24/11 4:27:31 PM3/24/11 4:27:31 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

190 ❘ CHAPTER 7 TRACKING QUALITY

Defi ning Test Cases

Each PBI needs test cases to ensure that the software is exercised suffi ciently to ensure that it meets its quality expectations. The more complex the feature, the more test cases you need to adequately test it. You should use negative testing, where you test for failure conditions, in addition to positive testing, where you test for success conditions.

Each item in the product backlog needs to be tested after it is scheduled in a sprint and implemented. A project with hundreds of PBIs will have hundreds of test cases. Some of them will be simple, and some will be complex; some will contain automated steps, and others will be automated.

There are a number of ways to create test cases, both from Visual Studio and from Microsoft Test Manager. With either tool, it ’ s important to link the test case back to the PBI. The following are the two methods for creating test cases:

Visual Studio — With a PBI open in Visual Studio, go to the Test Cases tab and click the New button. A dialog box appears, as shown in Figure 7 - 11, asking for very basic information. After you provide this information and click OK, Visual Studio opens the New Test Case window with the PBI linkage already populated.

➤

FIGURE 7 - 11: Creating a test case from a PBI in Visual Studio.

Microsoft Test Manager — Right - click Work Items in the Team Explorer and select New Work Item ➪ Test Case. If you use this method rather than creating the test case from the PBI, you have to manually link the two work items (the PBI and the test case).

You can link an existing PBI to an existing test case in Visual Studio from either item by using one of two methods:

From an existing PBI — Select the Test Cases tab in Visual Studio and click the Link To button. A dialog appears, and in it you can create or browse queries that will search for test case items.

From an existing test case — Select the Tested Backlog Items tab in Visual Studio and click the Link To button. A dialog appears, and in it you can create or browse queries that will search for PBIs.

➤

c07.indd 190c07.indd 190 3/24/11 4:27:32 PM3/24/11 4:27:32 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

Regardless of how you create a test case and link it to the PBI, you need to record some basic information about it, as shown in Table 7 - 1. Figure 7 - 12 shows the Visual Studio screen for entering a test case.

AT TRIBUTE DESCRIPTION

Title The name of the test case that appears on reports and queries.

Iteration The name of the project, release, or sprint to which this test case applies.

Area The component of the system to which this test case applies.

Assigned To The person responsible for defi ning the test case.

State The current state of the test case. It defaults to Design, meaning that the test

case must be designed. The next State setting in the workfl ow is Ready, which

means that the test is ready to be run. Finally, if the test is no longer valid, it can

be set to Closed.

Priority The relative priority of the test case, ranging from 1 to 4 and defaulting to 2.

A lower number indicates a higher priority.

Automation Status An indication of whether this is a manual or automated test. The default is

Not Automated. Another initial value is Planned, meaning that there will be

an automated test associated with this test case. Once you associate an

automation class with the test case, the status is Automated.

Description (on the

Summary tab)

A textual description of the test case.

TABLE 7-1: Essential Test Case Attributes

FIGURE 7 - 12: Entering a test case in Visual Studio.

Test Cases ❘ 191

c07.indd 191c07.indd 191 3/24/11 4:27:32 PM3/24/11 4:27:32 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

192 ❘ CHAPTER 7 TRACKING QUALITY

The Workfl ow for Test Cases

Figure 7 - 13 shows the workfl ow associated with a test case. When a test case is fi rst identifi ed, it is in the design state, meaning that the test must be designed and implemented. After a product owner or team member designs the test, the test case enters the ready state, meaning that the test is ready to be run. It remains in the ready state for the duration of the project, until the test is no longer needed, at which point it is placed in the closed state.

Test cases in the design state represent potential risk to the project. They represent PBIs that need to be tested but for which nobody has completely defi ned test cases. It may not be a very large risk, because the PBIs may not be built yet, but when they are built and ready for testing, additional work must be done before testing can begin.

To fi nd the list of test cases that are still in the design state, you simply run the predefi ned test cases query in the Current Sprint folder of Team Queries and then sort on the State fi eld. Figure 7 - 14 shows this query.

Design

Closed Ready

UpdatedDuplicate

Reactivated Completed

Duplicate

Reactivated

New test case

FIGURE 7 - 13: Test case workfl ow.

FIGURE 7 - 14: The test cases query.

It ’ s okay to have many test cases in the design state early in a release. But as the iterations progress, more test cases should be in the ready state. Figure 7 - 15 illustrates a theoretical project in which test

c07.indd 192c07.indd 192 3/24/11 4:27:33 PM3/24/11 4:27:33 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

Defi ning Manual Steps for a Test Case

In Visual Studio Ultimate Edition and Visual Studio Test Professional Edition, you can defi ne manual steps for a test. In other editions of Visual Studio, you must specify the test steps in the description of the test case. This section applies to Visual Studio Ultimate and Visual Studio Test Professional.

You enter manual steps for a test case into Test Manager. You can access the screen to enter manual steps in one of two ways:

From the test case in Visual Studio — Go to the Steps tab and click the Open for Edit button.

From Test Manager — Launch Test Manager from the Start menu in Windows. Open a test plan and click the Add button in the right side of the window. Test Manager brings up a query for all test cases not associated with the test plan. Choose the test case you want to add and click OK. Then open the test case.

Regardless of the method you use here, the result is the screen shown in Figure 7 - 16.

In addition to defi ning the manual steps, you can reference shared steps by clicking the Insert Shared Steps button. TFS launches a window and displays the shared steps, which you can include in this test case.

➤

FIGURE 7 - 15: A test case readiness report.

cases are created early and initialized in the design state and then marked as ready when they are fully defi ned with manual and automated steps.

Test Cases ❘ 193

c07.indd 193c07.indd 193 3/24/11 4:27:34 PM3/24/11 4:27:34 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

194 ❘ CHAPTER 7 TRACKING QUALITY

Defi ning Shared Steps for Test Cases

If you have a sequence of steps that is used in many test cases, you can save yourself some time by defi ning the sequence once and then referencing it from test cases. You store this sequence of steps in TFS as a shared steps work item. Doing this is a convenient way to ensure consistency across test cases.

FIGURE 7 - 16: Adding manual steps to a test case.

For example, in a public - facing web application, portions of the site might require authentication. In those parts of the site, the user must answer a challenge, such as providing the correct username and password, and then is granted access. This process might have the following three steps:

1. Click the Sign In link in the upper right of the screen.

2. Enter the username Shopper and the password RichGuy .

3. Click the Sign In button.

In a shared steps work item, you specify the sequence of steps that a team member will follow as part of validating a test case. For each step, you specify instructions for carrying out the step and the expected result. When the test is run later, the team member will be prompted to run each step and verify that the result is as expected.

Like manual steps, shared steps are available only in Visual Studio Ultimate and Visual Studio Test Professional, so this section applies only to those versions.

c07.indd 194c07.indd 194 3/24/11 4:27:34 PM3/24/11 4:27:34 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

You can create a shared steps defi nition in either Test Manager or Visual Studio, but you can only edit the steps within Test Manager. When using Visual Studio, you follow these steps:

1. Right - click Work Items in the Team Explorer and select New Work Item ➪ Shared Steps.

2. Fill in the fi elds. Only Title is required.

3. Click Save Work Item to save the shared step defi nition. If you don ’ t save the shared step defi nition, you won ’ t be able to edit it in Test Manager. Figure 7 - 17 shows the shared step created in Visual Studio before it is edited in Test Manager.

4. From the shared steps work item in Visual Studio (see Figure 7 - 17), click the Open for Edit button. The Test Manager screen appears, and you can edit the shared steps.

To create shared steps from Test Manager, follow these steps:

1. From a test case, select a set of steps by Shift+clicking the steps.

2. Right - click these steps and select Create Shared Step. The Test Manager Shared Steps edit screen appears, as shown in Figure 7 - 18.

FIGURE 7 - 17: A shared steps work item in Visual Studio.

In Figure 7 - 17, note that there are very few data elements captured. Shared steps are more of a productivity tool than a tracking element for Scrum, so they are not associated with iterations, PBIs, and so on.

Test Cases ❘ 195

c07.indd 195c07.indd 195 3/24/11 4:27:39 PM3/24/11 4:27:39 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

196 ❘ CHAPTER 7 TRACKING QUALITY

Using Automated Steps for Test Cases

Most components you build in Visual Studio will have automated tests. Some will have just a few automated tests, and others will have hundreds or thousands of them. Visual Studio has very sophisticated tools for creating automated tests — from simple unit tests that validate individual components or assemblies to coded user interface tests that verify onscreen data and actions to performance tests.

Component tests in Visual Studio are generally defi ned in code. They are code artifacts that exercise specifi c technical features of a system and validate those features against expected results. Visual Studio ships with templates where you replace the stubbed code with your own testing logic. The tests are attributed with [TestMethod] so they can be run as part of the Visual Studio testing infrastructure. The results of the tests are stored in the fi le system, either on the client if tests are executed there or on the build server if tests are queued and built centrally.

Manual tests, on the other hand, are generally stored in SQL and managed by TFS, not in code. They are accessible from the Team Explorer, from Visual Studio, and from Test Manager. The Test Manager tool that ships with Visual Studio Ultimate and Visual Studio Test Professional deals mostly with manual tests.

FIGURE 7 - 18: A shared steps work item in Test Manager.

You can also create a shared steps defi nition from Visual Studio or Test Manager. Regardless of how you create it, you must edit the actions and results in Test Manager.

c07.indd 196c07.indd 196 3/24/11 4:27:44 PM3/24/11 4:27:44 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

Automated steps are the intersection between the component - level testing infrastructure of Visual Studio and the system - level testing infrastructure of TFS. You defi ne component - level tests in Visual Studio and then reference those tests in TFS. When you run test scripts in Test Manager, Lab Manager communicates with System Center Virtual Machine Manager (VMM) to start a virtual machine to run the component - level tests.

Automated testing and Agile testing methodologies are beyond the scope of this book.

USING MICROSOFT TEST MANAGER TO DEFINE TEST PLANS

Now that you ’ ve looked at defi ning test cases in Visual Studio and shared steps across Visual Studio and Test Manager, you can examine Test Manager in more detail. Test Manager has four major sections:

Plan — In this section, you defi ne your test plans, test suites, and test cases. It has tools for linking test cases to PBIs and assigning them to testers. This is the main section that you ’ ll use for managing and tracking items for sprints.

Test — In this section, you run the tests. It works with Lab Manager to start virtual machines in which the tests are run. It is optimized for running test and verifying bugs.

Track — This is an alternate user interface to Visual Studio ’ s query screens and some build - management functions. In this section you can access all the personal and team queries and defi ne new ones.

Organize — This section provides an overview of the testing environment and has some reporting for showing the status of tests.

Figure 7 - 19 shows the startup screen of Microsoft Test Manager, showing a test plan titled Sprint 1. From the top menu, you can navigate to the four primary sections just described.

➤

FIGURE 7 - 19: Microsoft Test Manager.

Using Microsoft Test Manager to Defi ne Test Plans ❘ 197

c07.indd 197c07.indd 197 3/24/11 4:27:49 PM3/24/11 4:27:49 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

198 ❘ CHAPTER 7 TRACKING QUALITY

Test Manager provides tools for defi ning, organizing, and reporting on test plans. The goal of test planning is to group specifi c test cases into larger sets that can then be applied and measured at certain milestones within a project. Rather than track individual test cases, you can track the status of your test plan. Test Manager enables a team to organize test plans based on PBIs.

A test plan in Test Manager is a collection of test suites. A test suite is a collection of test cases. Test cases are used to verify the correctness of PBIs.

For a Scrum project, you can align test plans to sprints. At the beginning of a sprint, after the team has committed to PBIs in the sprint planning meeting, you should verify that you have test cases defi ned for each PBI. Then you can use Test Manager to defi ne a test plan for the sprint.

Figure 7 - 20 shows the hierarchy of test plans, test suites, and test cases. The following sections discuss test plans and test suites in more detail. For information on creating test cases, see the “ Defi ning Test Cases ” section earlier in this chapter.

You can defi ne test cases in Test Manager or in Visual Studio.

Test plan 1

�

1 � Test suite Test case

FIGURE 7 - 20: Test plans, test suites, and test cases.

From a data tracking perspective, Test Manager is a superset of Visual Studio. While Visual Studio enables tracking and reporting at the test case level, Test Manager enables grouping individual test cases or PBIs for sprint - level planning. Another distinction is that whereas Test Manager is more about tracking and running tests, Visual Studio is more focused on the engineering.

A test case lists the specifi c steps that the product owner or team members will follow to verify that a PBI meets expectations. Your team can defi ne test cases in either Test Manager or Visual Studio. You can link test cases to PBIs so that their status can be tracked together.

Test cases are not unique to Scrum, and most teams are familiar with test cases. Each PBI should have at least one test case, but most PBIs have dozens of them.

In Test Manager, you can group PBIs or test cases into test suites, which are selected via queries. The purpose of a test suite is to group test cases together so they can be tracked as a unit. A test suite is really nothing more than a collection of test cases.

By including test cases in test suites through queries, you can bring the test cases in from TFS automatically. A good starting point for this technique is to query by area path or sprint to create test suites that cover the entire area path or sprint, respectively.

c07.indd 198c07.indd 198 3/24/11 4:27:50 PM3/24/11 4:27:50 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

Organizing Test Plans

Test plans provide structure for organizing test cases based on PBIs. Each test plan typically corresponds to a sprint and encompasses all PBIs for the sprint. Alternatively, a test plan can organize test cases based on system function, such as security or performance, rather than correspond to a sprint. In that case, the test plan will span sprints because the PBIs are addressed in different sprints.

At the start of each sprint, a team should create a test plan that will be executed at the end of the sprint. Defi ning the test plan up front makes it clear exactly what will be tested. The test plan should include all of the tests associated with all the PBIs that are included in the sprint.

You create a test plan in Test Manager from the testing center. When Test Manager is launched, the dialog box shown in Figure 7 - 21 appears. You can use this dialog to add and delete test plans.

When a team creates a test plan, it defi nes the properties of the plan. The test plan properties include basic information about the test plan, such as when it starts and ends and its state. These attributes can be helpful in tracking progress with the test plan progress report. For instance, if a test plan is active, then its results in the test plan progress report are important. If the test plan is not active, you can fi lter out its results in the test plan progress report.

Figure 7 - 22 shows the properties of a test plan, which you access by clicking the Properties link at the top left of Test Manager. You can also access test plans from the Test Plan Manager link on the Organize tab.

FIGURE 7 - 21: Creating a test plan in Test Manager.

FIGURE 7 - 22: Test plan properties.

Using Microsoft Test Manager to Defi ne Test Plans ❘ 199

c07.indd 199c07.indd 199 3/24/11 4:27:55 PM3/24/11 4:27:55 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

200 ❘ CHAPTER 7 TRACKING QUALITY

More important than the properties, however, are the actual contents of the test plan. You access the test plan contents by clicking the Contents link in the Plan screen, which is available from the Plan tab in Test Manager. Figure 7 - 23 shows the Plan screen, which displays the test suites and test cases. Test suites are just a further grouping construct to organize PBIs or test cases.

FIGURE 7 - 23: The test plan Contents screen.

Organizing Test Suites

Using test suites is a convenient way to group test cases and the PBIs that they validate. For instance, using the area path to identify all PBIs that perform a common function is a natural grouping because the PBIs are related in the product. Filtering it down further, a test suite may encompass just those PBIs that are being implemented within a sprint.

You can add test suites in Test Manager by clicking the Add Requirements button on the test plan Contents screen (Figure 7 - 23). When you click the Add Requirements button, Test Manager opens a screen for selecting PBIs (see Figure 7 - 24). You can use this screen to specify a query for searching PBIs to include in the test suite. In this example, all PBIs under the Orders area path are being selected for inclusion in the test case. With the PBIs selected, you can click the Add Requirements to Plan button to add these PBIs to the test suite and to reference the test cases linked to those PBIs.

c07.indd 200c07.indd 200 3/24/11 4:27:56 PM3/24/11 4:27:56 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

Figure 7 - 25 shows a test plan named Sprint 1 with two test suites: Home Page Functions and Orders. To defi ne these test suites, you include each PBI within either the Home Page Functions or Orders area path. In this example, the fi rst PBI in Orders is named Order History Search, and it has four test cases.

FIGURE 7 - 24: Selecting items for a test suite.

The word requirements in Test Manager may not be the proper term for Scrum. Although its terminology isn ’ t Scrum specifi c, Test Manager is useful with any methodology, including Scrum.

In Agile development, requirements are captured as user stories. And in Scrum, user stories are tracked as PBIs. So, to organize a test plan around PBIs, you use the Add Requirements to Plan button. This way, you can add PBIs to the test suite and then add test cases to the test suite.

Using Microsoft Test Manager to Defi ne Test Plans ❘ 201

c07.indd 201c07.indd 201 3/24/11 4:27:57 PM3/24/11 4:27:57 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

202 ❘ CHAPTER 7 TRACKING QUALITY

SUMMARY

Great software has two common elements: usefulness and predictability. Test cases measure usefulness, verifying that the product performs the intended function and meets expectations. Bugs are items that must be fi xed within the product so it functions correctly and reliably. By fi nding and tracking bugs, you can measure predictability.

You count and track bugs in TFS. Your team can fi x bugs within the current sprint or defer them to a later sprint. In either case, the bugs are added to the backlog, where they’re estimated and classifi ed like other PBIs. After you commit to fi xing a bug, you create the task. Tasks track remaining work (in hours or days), as opposed to PBIs, which track relative effort (for example, in story points).

You count and track test cases in TFS by using a combination of Visual Studio and Test Manager. In Visual Studio, there are two built-in reports for tracking test cases: test plan readiness and test case progress. Test Manager enables you to organize test cases into test suites and test plans for greater control when handling thousands of individual tests. Test Manager also includes a sophisticated manual testing tool you can use to script and later automate manual tests. Team members can use this tool to execute manual tests and record whether the system passed or failed each test.

In Chapter 8, you’ll see how it all comes together in a release.

FIGURE 7 - 25: Test Manager test plan contents.

In a production environment, there may be hundreds of PBIs and thousands of test cases, so organization is quite important. This is where test suites come in handy.

c07.indd 202c07.indd 202 3/24/11 4:28:02 PM3/24/11 4:28:02 PM

C op

yr ig

ht ©

2 01

1. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed