E 2 635 harsha

ContentServer12.pdf

Home >Information Systems homework help >E 2 635 harsha

March 2014 (13:1) | MIS Quarterly Executive 15

MISQUarterly Executive

Many IT Projects Still Suffer From Poor Estimation1,2 Despite a great deal of attention in the trade and academic press, IT projects continue to fail

at an alarmingly high rate. One of the most-cited reasons for these failures is poor estimation practices.

“Unrealistic expectations based on inaccurate estimates are the single largest cause of [IT project] failure.”3

Estimation is defined as an informed assessment of an uncertain event. For IT project managers, accurate estimates are the foundation for effective project planning and execution and, ultimately, project success. Unfortunately, most project managers do a very poor job of estimating and, as a result, most IT projects are classified as failures—61% in the latest Standish Group report.4 The Standish Group’s data shows some improvement in the overall success rate since 2004 (which it partially attributes to the Agile development process and improved project management expertise). However, its figures show a slight increase in both time and cost overruns since 2010—signaling that there is still much room for improvement. According to Standish, 74% of challenged projects experience time overruns and 59%

1 Leslie Willcocks is the accepting senior editor for this article. 2 The authors would like to thank Steve McConnell and Arin Sime for their input throughout this research project. We would also like to acknowledge the McIntire School of Commerce for providing financial support for this research project, thank research assistant Meg Raymond and thank the Project Management Institute for posting a link to our survey. 3 Futrell, R. T., Shafer, D. F. and Shafer, L. I. Quality Software Project Management, Prentice Hall, 2002. 4 The Standish Group: Chaos Manifesto 2013, available at http://versionone.com/assets/img/files/ChaosManifesto2013.pdf.

IT Project Estimation: Contemporary Practices and Management Guidelines Many IT projects continue to suffer from poor estimation. Indeed, the accuracy of estimation has hardly changed from that reported in a seminal study carried out over 20 years ago. Based on findings from two recent survey-based studies, which replicated and then extended the original study, we provide guidelines for improving IT project estimation, taking account of the greater use today of Agile, rather than traditional Waterfall, development methods.1,2

R. Ryan Nelson University of Virginia (U. S.)

Michael G. Morris

IT Project Estimation

experience cost overruns. Further evidence can be found in a review of 180 IT projects completed between 1999-2013,5 64% of which suffered from poor estimation.

In this article, we examine the practice of IT project estimation, report the findings from two studies and provide recommendations to help project managers improve project estimation.

“An estimate is the most optimistic prediction that has a non-zero probability of coming true … Accepting this definition leads irrevocably toward a method called what’s-the-earliest-date-by-which- you-can’t-prove-you-won’t-be-finished estimating.”6

Why Estimates Matter Accurate estimation is very important to IT

project managers for a wide variety of reasons: ● Avoiding the vicious cycle: poor estimation

results in more schedule pressure, which in turn creates more stress, producing more mistakes, and ultimately more schedule slips, creating even more schedule pressure, and so on.7

● Avoiding the ripple effect: slippage in one project’s completion date can have a ripple effect on other projects and stakeholders throughout the organization. Effective organizations require a portfolio of well- coordinated projects to succeed, and better estimation leads to better coordination with other tasks and projects throughout the portfolio.8

● Avoiding late-in-the-project discovery that the project has been underestimated,

5 See Nelson, R. R. “Project Retrospectives: Evaluating Project Success, Failure, and Everything in Between,” MIS Quarterly Executive (4:3), 2005, pp. 361-372; Nelson, R. R. “IT Project Management: Infamous Failures, Classic Mistakes, and Best Practices,” MIS Quarterly Executive (6:2), 2007, pp. 67-78; and Nelson, R. R. and Jansen, K. J. “Mapping and Managing Momentum in IT Projects,” MIS Quarterly Executive (8:3), 2009, pp. 141-148. 6 DeMarco, T. Controlling Software Projects: Management, Measurement and Estimation, Yourdon Press, 1982. 7 McConnell, S. Software Estimation: Demystifying the Black Art, Microsoft Press, 2006. 8 Boehm, B. Software Engineering Economics, Prentice-Hall, 1981. This book contains a thorough discussion of software estimation and scheduling, which is presented in terms of Boehm’s COCOMO cost- estimation model.

which is difficult, if not impossible to correct.9

● Facilitating better budgeting: accurate budgets depend on accurate estimates of size, effort and time.10

● Evaluating project personnel: compensation is often tied to project performance.

● Generating more credibility for the project team: being part of a team that brings a project in on budget and schedule can do wonders for an individual’s career, not to mention job satisfaction.11

In practice, estimation is both an art and a science. While the art of estimation is closely tied to personal experience, heuristics and common sense, the science of estimation can be based on complex statistical analysis, algorithms, historical data and computer-based tools.

To gain a better understanding of exactly how these various methods are used in practice, we conducted two research studies, focusing on contemporary estimation practices and their linkage to project success. The first study focused on an organizational-level analysis of project estimation. Results from 60 IT professionals suggested that, consistent with responses from a similar study carried out 20 years ago, there is still much room for improvement in how estimates are generated and used within organizations. Moving down a level of analysis, the second study focused on project-level data from 115 IT professionals. It also compared traditional Waterfall and Agile development methods with respect to estimation practices and project success. (The survey methodologies used for each of these studies are described in more detail in the Appendix.)

Research Study 1: Organization-Level Analysis of

Project Estimation In 1992, Lederer and Prasad published the

results of a survey of 115 IT professionals on their

9 Brooks, F. The Mythical Man-Month, Addison-Wesley, 1975. 10 Jones, C. Estimating Software Costs, McGraw-Hill, 1998. 11 DeMarco, T. and Lister, T. Peopleware: Productive Projects and Teams (3rd Edition), Addison-Wesley, 2013.

March 2014 (13:1) | MIS Quarterly Executive 17

IT Project Estimation: Contemporary Practices and Management Guidelines

organizations’ cost estimation practices,12 which we have used as a baseline for our research. That study provided an in-depth understanding of the estimation process in the early 1990s— an era of organizational computing marked by the rapid diffusion of the personal computer, the rise of network computing (before the takeoff of the Internet) and the beginning of the business process reengineering movement. Lederer and Prasad’s research exposed a stark contrast between the perceived importance of accurate project estimation and the state of practice, and they prescribed a set of management guidelines aimed at improving estimation practices.

The primary objective of our first study was to discover how project estimation perceptions and practices have changed over the more than 20 years since Lederer and Prasad’s original study. We modified the survey used by Lederer and Prasad and distributed it to a similar target population.13 The results of our study are compared and contrasted with the Lederer and Prasad findings below.

There is Still a Significant Need for Estimation Improvement

After 20 years, there is still a profound need for improvement in IT project estimation—as highlighted in Table 1. Although the vast majority of our respondents believe that estimation is important (88% described it as “very important”

12 Lederer, A. L. and Prasad, J. “Nine Management Guidelines for Better Cost Estimating,” Communications of the ACM (35:2), 1992, pp. 51-59. This was one of Lederer and Prasad’s original studies on cost estimation, with other informative articles being published over a 10-year period. 13 While the general profile of respondents and companies in our study is similar to the Lederer and Prasad sample, there was a significant difference in industries represented—with their largest percentage of respondents coming from manufacturing (32%), compared with service industries (31%) in our sample.

or “moderately important,” the highest two possible ratings on the five-point scale), the data also suggests that only 31% of projects are completed reasonably close to their estimated time or cost. The good news, however, is that our study shows that both estimation awareness and performance have improved somewhat since the early 1990s.

How Estimates Are Used As stated above, estimates can be important

for a variety of reasons. Table 2 shows that estimation seems to be used more for project planning and control (rows 1 to 5) than for evaluating estimators, developers and others involved in a project (rows 6 to 8). These findings are consistent with Lederer and Prasad’s findings, with the top five remaining the same, albeit in a different rank order.

Our respondents were then asked about their organizations’ most common estimation practices, and the top five responses are listed in Table 3. Interestingly, although 75% of projects use formal estimates, only 54% involved developers in the estimation process—suggesting that 21% of projects do not have any estimation input from the developers even though they use formal estimates. This finding is contrary to the first of Lederer and Prasad’s “Nine Management Guidelines for Better Cost Estimating:” “Assign the initial estimating task to the final developer.” A second Lederer and Prasad guideline is “Monitor the course of a project from the preparation of the estimate through project completion.” This practice was reported for 71% of projects in our study, which is virtually identical to the 70% reported by Lederer and Prasad in 1992.

Table 1: Importance and Accuracy of IT Project Estimation Lederer &

Prasad Study 1

The accurate estimation of IT projects is moderately/very important. 84% 88%

What percentage of all IT projects significantly overrun their estimates? 63% 56%

What percentage of all IT projects significantly underrun their estimates? 14% 13%

What percentage of all IT projects are completed at cost close to estimate. 23% 31%

IT Project Estimation

The Estimation Process and Causes of Inaccuracies

Respondents were asked about the basis they used for their estimations. As seen in Table 4, they rely most heavily on comparisons to similar, past projects when arriving at their estimates— whether these comparisons are based on personal memory or on documented facts. It is interesting to note that the rank ordering of the bases by our respondents is virtually identical to what Lederer and Prasad found in their research 20 years ago (only items 4 and 5 were reversed in order).

Our survey included a question on peer or team-reviewed estimates, which was not included in the original Lederer and Prasad survey, because this practice is promoted by the use of Agile development. Our respondents reported that this practice was the third-most extensively used basis for estimation.

The final part of our study replicating the Lederer and Prasad study was to ask respondents to identify the factors most responsible for inaccurate estimates—giving them a total of 26 causes to choose from (the top five are listed in Table 5). Again, these findings are remarkably consistent with Lederer & Prasad’s (the notable exception being that “Insufficient user-analyst communication and understanding” dropped from fourth position in their study to ninth in ours).

The clear message continues to be that initial estimates will be inaccurate because of frequent changes by users, lack of users’ understanding of requirements, inadequate task/problem identification and insufficient analysis at the estimation stage. As a result, estimators are encouraged to revise estimates during the course of the project (reported by 63% of our respondent organizations) and to delay committing to a target for as long as possible.

Table 2: How Estimates are Used

Use of Estimate Importance of Use

(1 = Very unimportant; 5 = Very important)

1. To select proposed projects for implementation 4.40

2. To staff projects 4.20

3. To schedule projects 4.17

4. To quote the charges to users for projects 4.04

5. To control or monitor project implementation 3.93

6. To evaluate project estimators 3.26

7. To audit project success 3.33

8. To evaluate project developers 3.13

Table 3: Project Estimation Practices

Top 5 Responses Percentage of IT

Projects

1. Formal estimates are prepared 75%

2. Formal monitoring of project progress 71%

3. Estimate revised because requirements change 63%

4. Cost-benefit analysis used to justify project 61%

5. Developers participate in estimation 54%

March 2014 (13:1) | MIS Quarterly Executive 19

IT Project Estimation: Contemporary Practices and Management Guidelines

Research Study 2: Project-Level Analysis of Project Estimation

Study 1 enabled us to gain an overall view of organizational project estimation practices. We wanted to supplement these findings with a more detailed second study of project estimation practices that focused on project-level data. In Study 2, we therefore surveyed 115 IT professionals and asked them to respond with data about their most recently completed IT projects.

Another factor we wanted to explore in Study 2 (based on what we learned in Study 1 as well as input from two software estimation experts consulted throughout the research project) is the

changes in the IT development landscape since Lederer and Prasad’s 1992 study. Specifically, the growth of Agile development has fundamentally altered the way in which projects are conceived and managed. Collecting data at the project level allowed us to learn more about the usefulness and accuracy of project-estimation practices in the context of specific development environments (e.g., Agile vs. traditional Waterfall development), which obviously carries important implications for IT managers today.

As shown in Table 6, we found that traditional Waterfall methodologies (or variants) were still the most widely used. However, Agile methodologies made up a significant proportion

Table 4: Bases of the Estimating Process

Basis Extensiveness of Use

(1 = Not used at all; 5 = Extremely extensive)

1. Comparison to similar, past projects based on personal memory 3.48

2. Comparison to similar, past projects based on documented facts 3.45

3. Peer or team-reviewed estimates 3.08

4. A simple arithmetic formula (such as summing task durations) 2.96

5. Intuition 2.94

6. Guessing 2.76

7. Established standards (such as averages, standard deviations, etc.)

2.71

8. An estimation tool (e.g., software package) 1.63

9. A complex statistical formula (such as multiple regression, differential equations, etc.)

1.57

Table 5: Causes of Inaccurate Estimates

Causes (Top 5 Out of 26) Extent of Responsibility

(1 = Not resp. at all; 5 = Extremely responsible)

1. Frequent user requests for changes 3.70

2. Users’ lack of understanding of their own requirements 3.54

3. Overlooked tasks 3.35

4. Poor or imprecise problem definition 3.26

5. Insufficient analysis when developing estimate 3.11

IT Project Estimation

(nearly a third) of all projects in our Study 2 sample. 14

Consistent with the findings of Study 1, the 94 projects in Study 2 for which estimated vs. actual costs were reported tended to come in over budget, although there was also a healthy number of projects on or even under budget in the sample (see Table 7).

Clearly, the 40% of projects reported as over budget indicates that estimation and/ or execution problems persist in IT projects. However, the 60% of projects that are on or under budget is a hopeful sign that the project management discipline is progressing. Moreover, Agile projects seem to be performing better than Waterfall projects in terms of cost performance against budget. Of the 38 (out of 94) projects that were over budget, the average percentage over the original estimate was nearly 73%, with a range between 1% and 900%. For the 29 projects under budget, the range was much smaller, from 2% to 93%, with an average of 17%.

14 Schwaber, K. and Beedle, M. Agile software development with Scrum, Prentice Hall, 2002.

In terms of schedule compared to the original estimate, on average, projects started 25 days late (with none starting early), and finished an average of 56 days late (with a maximum of 3.1 years). Once again, Agile projects tended to start and finish closer to their estimated times. Table 8 summarizes the results for project timelines.

In terms of functionality compared to the original estimate, regardless of start or finish time, 90% of the originally specified requirements were completed.

A Detailed Look at Project-Level Estimation Methods and Practices

Based on the responses in Study 1 and input from our two experts, we constructed a list of commonly accepted estimation best practices and asked respondents in Study 2 to say which of them were followed during their most recently completed IT projects (see Table 9). The good news is that several of these practices are used on projects in the organizations we sampled. Our survey showed that Waterfall and Agile projects employ similar estimation practices, with the exception of preparing formal estimates, which,

Table 6: Development Methodologies Methodology Count Percentage

Waterfall (or variant) 45 47.9%

Agile—Scrum14 11 11.7%

Agile—other 19 20.2%

Other (e.g., ad hoc and hybrid) 19 20.2%

Table 7: Cost Performance Estimates vs. Actuals Total Waterfall Agile

Over budget 40% 47% 33%

Under budget 31% 33% 33%

On budget 29% 20% 33%

Table 8: Schedule Performance Start Time Finish Time

Total WF Agile Total WF Agile

Late 36% 31% 30% 61% 67% 53%

Early 0% 0% 0% 17% 13% 13%

On Time 64% 69% 70% 22% 20% 33%

March 2014 (13:1) | MIS Quarterly Executive 21

IT Project Estimation: Contemporary Practices and Management Guidelines

not surprisingly, is more prevalent in Waterfall projects.

Table 10 lists the 11 most commonly used methods for estimating the costs of respondents’ most recent IT projects.

The rankings in Table 10 compare favorably with the organizational-level data from Study 1 (see Table 4). (Note, however, that due to the specific project focus of Study 2 the question was asked differently.) The top three cost-estimation methods were identical in both studies. Other common—and surprising—findings are the apparently wide use of “guessing” as a common basis of cost estimation (particularly in Agile projects) and the relatively low use of estimation tools to support the process (although the use of tools has increased slightly from the 17%

reported by Lederer and Prasad in 1992). Cost-estimation tools most commonly cited by respondents were Excel, internally developed proprietary tools and Construx Estimate (a freeware tool), although none of these was widely used.

Table 10 indicates some differences in the cost-estimation methods used for Waterfall and Agile projects. Whereas Waterfall projects tend to compare with past projects, use individually prepared and reviewed estimates, and use established organizational standards, Agile projects are more likely to rely on expert judgment, formulas and group-based estimates.

To provide additional detail about the estimation process and bases for estimation, respondents also identified the most common

Table 9: Estimation Best Practices In Use Best Practice Total WF Agile

A formal estimate was prepared 86% 93% 73%

Progress against the estimate was formally monitored throughout the project 80% 82% 80%

The same people who eventually executed the project plan (e.g., developers) also participated in the preparation of the initial estimate 67% 69% 63%

The estimate was revised to accompany changes in user requirements during the project 61% 64% 57%

Project scope was reduced to meet a target completion date 38% 38% 40%

Table 10: Method for Estimating Project Cost Most- to Least-used Method Total WF Agile

Comparison to similar, past projects based on documented facts 59% 67% 43%

Comparison to similar, past projects based on personal memory 58% 58% 57%

Estimates created by individuals and reviewed in a group setting 56% 58% 57%

Expert judgment without use of documented facts 54% 51% 67%

A simple arithmetic formula (such as summing task durations) 52% 47% 57%

Estimates created in a group setting 49% 38% 70%

Estimates created by individuals and reviewed by individuals 45% 51% 37%

Guessing 39% 31% 47%

Established organizational standards (such as averages, standard deviations, etc.) 34% 51% 20%

An estimation tool (e.g., software package) 21% 27% 13%

A complex statistical formula (such as multiple regression) 9% 4% 20%

IT Project Estimation

methods used to estimate project size and complexity. Table 11 lists the five most common methods used across the large sample of Study 2 projects.

Work breakdown structure is by far the most commonly used method for estimating project size and complexity, which suggests a formality and structure that has developed over time in project management. However, the most surprising finding is the low level of use of lines of code for estimating size and complexity, despite prior studies and popular literature on software estimation reporting a heavy reliance on this method.15 There are two reasons why this method is now less popular than in the past. First, is the type of project typical in today’s IT environments. Increasingly, contemporary projects include a large number of integration or COTS (commercial off-the-shelf ) projects, which often make the use of lines of code challenging or impossible. The other contextual change includes the growth of Agile development methodologies, which often promote the use of relative size

15 See for example McConnell, S., op. cit., 2009 p. 198.

estimates (e.g., story points in Table 11). Our survey results suggest it may be necessary to reconsider and refine techniques promoted in the popular IT development literature concerned with the use (and even relevance of ) lines of code to estimate many of today’s IT projects.

Estimate Creation and Presentation In addition to the methods and practices used

to create IT project estimates, we also wanted to examine who was responsible for creating an estimate, when and how it was created and the form in which it was presented. Table 12 lists the person(s) most responsible for estimate creation.

These findings were as expected and similar for both Waterfall and Agile projects, with the exception that Agile projects were more likely to have the people doing the work (but not the technical lead) responsible for creating the estimates. Estimation experts recommend this as a best practice.

The only surprising result in Table 12 is the relatively large percentage of respondents reporting that someone outside of the project team (i.e., not directly involved with the

Table 11: Methods Used to Estimate Size and Complexity of IT Projects Most- to Least-used Method Total WF Agile

Work breakdown structure 71% 73% 53%

Work drivers (number of interface changes, number of new modules, etc.)

46% 58% 33%

Story points (relative estimate of the size and complexity of work in an Agile project)

32% 16% 67%

Function points (formal count of number of user inputs, outputs, inquiries, files and external interfaces based on IFPUG standards or other industry standard approach)

27% 36% 20%

Lines of code 8% 7% 7%

Table 12: Responsibility for Estimate Creation Person(s) Responsible (Multiple Responses Allowed) Total WF Agile

Technical lead 74% 73% 70%

Person(s) doing the work (not technical lead) 72% 64% 83%

Manager 53% 58% 43%

Independent estimator 14% 16% 13%

Someone else outside of the project team 20% 22% 17%

March 2014 (13:1) | MIS Quarterly Executive 23

IT Project Estimation: Contemporary Practices and Management Guidelines

project), but not an independent estimator, was responsible for creating the estimate. The open-ended comments provided with these responses suggested that people with little formal (or informal) estimation expertise, or even familiarity with the technology, are frequently driving project estimates. Amazingly, one respondent reported that the sales manager was responsible for creating the estimate for her most recent project.

To get a sense for the timing of estimate creation, we asked respondents when the project estimate was created. As expected, by far the most common response was that estimates were created before any work on the project was done (see Table 13). Note that a relatively high number of Agile projects (40%) were still creating estimates through product-concept-completion. Again, this is often viewed as a recommended practice.

Additional qualitative data gathered from respondents suggested that baseline estimates are commonly prepared at the proposal stage and updated as major milestones are achieved. In fact, one respondent indicated that estimates were refined at eight different points during his last IT project. Revisiting estimates throughout a

project is often a recommended best practice, so greater use of this practice could well improve estimation accuracy.

Finally, respondents reported a variety of techniques for presenting estimates to internal management or external clients. Rather than using a monolithic, single-point estimate, our survey indicates wide variation in how estimates are presented, including some differences between Waterfall and Agile projects (see Table 14).

It is surprising (and somewhat encouraging) that the use of a single-point estimate falls in the middle of Table 14, with other more contextually driven factors (ranges, feature lists or confidence factors), exceeding or rivaling the popularity of less flexible, single-point estimates.

To summarize, most of the 115 projects reported on in Study 2 were either late and/or over budget, with Agile projects faring somewhat better in both areas. A possible explanation for the better performance of Agile projects (in terms of estimation) is that they tend to rely more on expert judgment, formulas and group- based estimates. In addition, estimates for the Agile projects in the survey tended to be:

Table 13: Timing for Estimate Creation When Estimate is Created (Multiple Responses Allowed) Total WF Agile

Before any work on the project was done 72% 67% 73%

Requirements complete 46% 42% 30%

Through product-concept-completion 32% 29% 40%

At a defined gate or milestone in a stage-gate process 29% 33% 23%

Table 14: Estimate Presentation Most Common to Least Used (Multiple Responses Allowed) Total WF Agile

Cost or effort range 54% 56% 50%

Schedule range 49% 40% 53%

Feature list (e.g., definitely, probably or maybe) 49% 44% 60%

Single-point number 45% 47% 33%

Confidence factors (probability of delivering on-time and/or on-budget) 33% 36% 27%

Cases (e.g., best, most likely, worst) 29% 33% 17%

Plus-or-minus qualifiers (e.g., 6 months, +3 months, - 2 months) 26% 24% 23%

IT Project Estimation

● Prepared by people who would also be doing the work

● Baselined at the beginning of the project and updated throughout the project

● Presented in (cost/effort/schedule) ranges rather than as single-point numbers.

Common Estimation Problems In light of the cost, schedule and functionality

estimation challenges reported above, we also asked respondents which estimation-related problems they encountered during their last IT projects. They cited the following (listed in descending order of influence as reported in prior studies):16

1. Insufficient analysis when developing the estimate

2. Lack of adequate methodology/guidelines for estimation

3. Lack of historical data on past estimates and actual results

4. Pressure from managers or other stakeholders to increase or reduce estimates, and over-reliance on a single person’s estimate

5. A lack of visibility or control over actual data compared to estimates.

Although less directly linked to estimation per se, other problems that are either antecedents or consequences of poor estimation practice identified by managers included:

● Frequent user requests for changes ● Users’ lack of understanding of their own

requirements ● Overlooked tasks ● Changing personnel ● Poor or imprecise problem definition. Qualitative data provided by project

managers also suggested that drastic budget cuts by government agencies was a factor in causing projects to miss the original estimates. A representative comment was “The program was tracking in accordance with the estimate

16 The ranked order of 26 proposed causes of inaccurate estimates from Study 1 and Lederer and Prasad (1992).

both in terms of time and budget. The Federal agency simply slashed the budget.” Other inputs from respondents implied difficulties in managing offshore development teams (what one person termed “misinterpretation and rogue creativity” of team inputs), and undue pressure from stakeholders, including variations of the following comment: “Every project we estimate gets significant pushback from [stakeholders] to cut the estimate.”

Evaluation of Project Success Finally, we asked Study 2 respondents to

evaluate seven project success criteria for their most recent IT projects.17 As reported above, most (61%) projects were late, and 40% were over budget. While these numbers tell a similar story of project failure to that reported by the Standish Group, other criteria tell a quite different story. As demonstrated in Figure 1 and Table 15, the vast majority of projects in our survey met requirements, produced products/services that were used by their target constituencies, resulted in organizational learning and provided value. Moreover, overall stakeholder satisfaction was rated at 91%.

These findings suggest that roughly 9 out of 10 projects can be classified as “successful failures”—that is, projects that fail on one or more process criteria (schedule and/or budget in particular), but succeed on all of the important outcome criteria. Representative comments from the respondents included the following:

“We delivered 10% more scope with 10% less budget three weeks late.”

“The project came in slightly over schedule but exceeded expectations.”

“The board of directors had the project audited this month by external auditors and it is being showcased in our industry groups, newspapers and to our shareholders to demonstrate our capabilities.”

It is also interesting to compare our survey results with the benchmark results of 180 project retrospectives conducted in 130 different organizations between 1999 and 2013. Once

17 For a more detailed description of project success, see Nelson, R. R., op. cit., 2005, pp. 361-372.

March 2014 (13:1) | MIS Quarterly Executive 25

IT Project Estimation: Contemporary Practices and Management Guidelines

again, these collective findings suggest that IT projects tend to come in over schedule and/or budget, yet eventually meet requirements and add value to the organization. So, the bad news is that most IT projects continue to fail in terms of meeting estimates, but the good news is that stakeholders seem to be satisfied as long as the project results in a product/service that adds value within their organization.18

As previously noted, the most common determinant of project failure is poor estimation. Data suggests that this “classic mistake” is the most common of all, occurring in 64% of the 180 benchmark projects. In all of the studies reported in this article, there are many possible causes

18 Updated data from Nelson, R. R., op. cit., 2005, on 180 project retrospectives conducted in 130 different organizations between 1999 and 2013.

of poor estimation, including poor problem/ requirements definition, scope creep, lack of requisite knowledge and skills, inadequate estimation methodologies, lack of historical data, lack of visibility or control, and pressure from stakeholders. We now provide guidelines for addressing each of these problem areas.

Guidelines for Improving IT Project Estimation

Based on the findings of our research, we provide 10 guidelines for project managers on how to improve their estimation practices.

1. Admit You Have a Problem One of the striking revelations from our two

studies is that some of the key findings are very

Figure 1: Project Success Criteria OVERALL

STAKEHOLDER SATISFACTION

(91%)

LEARNING (87%)

USE (94%)

OUTCOME VALUE (89%)

COST (60%)

TIME (39%)

PRODUCT (90%)

PROCESS

Table 15: Project Success Criteria Yes Benchmark18

The project came in on schedule 39% 40%

The project came in on budget 60% 45%

The project produced a product of acceptable quality and met other product-related specifications, including requirements, usability, ease of use, modifiability and maintainability 90% 68%

The project’s resulting product/service is being used by its target constituencies 94% 88%

The project increased stakeholder knowledge and helped prepare the organization for future challenges 87% 82%

The project will directly result in improved efficiency and/or effectiveness for the client organization(s) 89% 70%

Overall, stakeholders were satisfied with the project 91% NA

IT Project Estimation

similar to those of Lederer and Prasad over 20 years ago. While there have been tremendous advancements in the project management discipline, technology, IT development processes and the education level of IT professionals, it is clear that developing accurate estimates continues to be a huge problem. As one respondent stated: “Our estimation process has been ‘spotty’ at best. … we’ve certainly not found the holy grail of estimating (yet).”

2. Revisit the Prescriptions Offered by Lederer and Prasad

Although Lederer and Prasad’s work was done over 20 years ago, many of their prescriptions can be adapted for today’s IT development environment, even though that environment has changed (e.g., the increased use of Agile methods). Based on our studies, several of their specific recommendations are still valid:

● Assign the initial estimating task to the final developers

● Delay finalizing the estimate until the end of a thorough study of requirements

● Monitor project progress closely— including actuals versus estimates

● Rely on documented facts, standards and simple arithmetic formulas rather than guessing, intuition, personal memory or complex formulas to generate the estimate.

3. Conduct Project Retrospectives. Project teams should be encouraged to

perform status reviews throughout the life of a project. Such reviews can happen, for example, at the end of each day (daily standup), two- week intervals, completion of a major milestone and project completion. In addition to learning lessons and making necessary adjustments, actual data can be captured for use in estimating the next sprint (the basic unit of development in Scrum), phase or project.

4. Employ Estimation Tools As shown in Tables 4 and 10 above, the use

of estimation tools (e.g., software packages) was relatively low in both of our studies (only 21% of Study 2 respondents reported using an estimation tool). Estimation tools can help to:

● Evaluate and sanity-check project plan alternatives against industry data or an organization’s historical data (if captured via project retrospectives)

● Negotiate a reasonable schedule and budget, using a tool’s reporting capability

● Improve visibility and control over actual data compared to estimates

● Coordinate estimated and actual project data within and across project teams.

In addition to Microsoft Project and Excel (including templates and add-ons), dedicated estimation tools include freeware19 and other packages.20

5. Understand Agile Approaches and Adapt Estimation Techniques Accordingly

The findings from our studies underscore the importance of Agile development methods in modern software development. Therefore, it is critically important that project managers understand these methods and how traditional estimation techniques can be adapted to fit the context of Agile development.21 Key principles behind the Agile Manifesto22 include:

● Our highest priority is to satisfy the customer through early and continuous delivery of valuable software

● Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale

● Working software is the primary measure of progress.

Clearly, frequent estimation and delivery cycles are central to the Agile approach—and an effective way to improve estimation (one sprint or deliverable at a time).

19 Freeware estimation tools include Construx Estimate (http:// www.construx.com/estimate) and COCOMO (http://sunset.usc.edu/ csse/research/COCOMOII). 20 Other estimation tools include QSM SLIM-Estimate (http:// www.qsm.com/tools/slim-estimate) and SEER (http://www.galorath. com). 21 For a good primer that explains the philosophy, tools and techniques associated with estimation for Agile development, see Cohn, M. Agile Estimating and Planning, Prentice Hall, 2005. Agile estimation methods include relative (a.k.a. story point) estimation and planning poker. 22 http://agilemanifesto.org/principles.html.

March 2014 (13:1) | MIS Quarterly Executive 27

IT Project Estimation: Contemporary Practices and Management Guidelines

6. Avoid Cognitive Biases Toward Estimation

People are subject to an almost limitless set of biases when making subjective judgments or decisions, and the process of estimation is no exception. Perhaps the best way to avoid letting biases creep into estimation judgments is to be aware of them. Some specific biases that are particularly dangerous for project managers in estimating projects include the following.

Underestimation Bias. People have a strong tendency to underestimate, particularly when the focal object of estimation is unknown or complex. This obviously includes software and/ or outsourced projects (and is worse when requirements, technologies, etc. are new or unknown). Indeed, in over 15 years of teaching estimation to mid-career IT professionals (average work experience more than 12 years), we have found that perhaps the most common tendency from in-class workshops is for them to underestimate when asked to give a range estimate for a series of questions.

The Planning Fallacy. Related to the underestimation bias, the planning fallacy23 is the tendency for people and organizations to underestimate the time, costs and risks of future actions and at the same time overestimate the benefits of those actions. Thus, the planning fallacy results not only in time overruns, but also in cost overruns and benefit shortfalls. Kahneman and Tversky’s original explanation for the fallacy was that planners focus on the most optimistic scenario for the task, rather than using their full experience of how much time similar tasks require. It is interesting to note that this bias only affects predictions about one’s own tasks; when uninvolved observers predict task completion times, they show a pessimistic bias, overestimating the time that will be taken.24 This suggests the need to involve external auditors and/or use peer- or team-reviewed estimates,

23 The term “planning fallacy” was first coined in 1979 in Kahneman, D. and Tversky, A. “Intuitive prediction: Biases and corrective procedures,” TIMS Studies in Management Science, (12), 1979, pp. 313-327. Since then the effect has been found for predictions of a wide variety of tasks, including tax form completion, school work, furniture assembly, computer programming and even origami. 24 Buehler, R., Griffin, D. and Ross, M. “Inside the Planning Fallacy: The causes and Consequences of Optimistic Time Predictions,” Gilovich, T., Griffin, D. and Kahneman, D. (Eds.) Heuristics and Biases: The Psychology of Intuitive Judgment, Cambridge University Press, 2002, pp. 250-270.

as our research shows that many firms are now doing.

Wishful Thinking. As implied by the results of our studies, project managers think tasks will be finished quickly and easily because that is what they want to happen. Unfortunately, there is a bigger penalty for underestimating (non-linear impact due to planning errors, shortchanged requirements, high-risk practices, etc.) than for overestimating (linear impact due to Parkinson’s Law, where work tends to expand to fill the estimate).

Self-Serving Bias. By taking credit for tasks that went well but blaming delays on outside influences, project managers or those centrally involved with creating project estimates often discount past evidence of how long a task should take. Experiments have demonstrated that when people made their predictions anonymously, they do not show an “optimistic bias.”25 This suggests that people sometimes make optimistic estimates to create a favorable impression with others.

Focalism Bias. The focalism bias refers to the mental discounting of factors believed to be outside the specifics of the project.26 These factors include “overhead” tasks (e.g., meetings, sickness and vacations) and “black swan events” (low-probability, high-impact risks).

7. Understand the Estimate- Convergence Graph

The estimate-convergence graph (a.k.a. the “cone of uncertainty”) derives from research that found that project estimates fall within predictable ranges at various stages of a project.27 One of our survey respondents reported on his company’s practice of requiring project managers to estimate in ranges. At the beginning of the feasibility phase, project managers are asked to come up with a 100% cushion, at the beginning of the definition phase, a 75% cushion, at the beginning of the design phase, a 50% cushion and at the beginning of the construction phase, a 25% cushion. Project

25 Pezzo, M., Litman, J. and Pezzo, S. “On the distinction between yuppies and hippies: Individual differences in prediction biases for planning future tasks,” Personality and Individual Differences (41:7), 2006, pp. 1359-1371. 26 Wilson, T., Wheatley, T., Meyers, J., Gilbert, D. and Axsom, D. “Focalism: A source of durability bias in affective forecasting,” Journal of Personality and Social Psychology (78:5), 2000, pp. 821-36. 27 Boehm, B., op. cit., 1981.

IT Project Estimation

managers update their estimates at the end of each phase, resulting in improved estimation accuracy as a project progresses through its life cycle.

8. Understand the Difference Between a Target and an Estimate

Estimates are arrived at based on careful analysis, whereas targets represent a desired schedule or cost and can be set without any analysis. Project managers should be encouraged to carefully set targets within a reasonable range of an estimate and delay making a firm commitment for as long as possible.

“Estimation’s role is to determine whether you have a realistic chance of meeting your targets. If the target is within roughly 20% of the estimated outcome, you can control the project to meet the targets (make minor adjustments to features, schedule and resources). If the gap between the target and the estimate is greater than about 20%, it is not possible to control the project to meet the targets.”28

9. Manage Stakeholders Given the role that stakeholders play in

defining project requirements, setting targets, changing requirements (i.e., scope creep) and exerting pressure on the project team throughout a project, project teams need to find more effective ways to involve stakeholders. For example, another key principle of Agile development is that “business people and developers must work together daily throughout the project.” This practice enhances transparency and may help control stakeholder pressure.

10. Develop Estimation Competencies of Project Managers and Team Members

As emphasized by the first guideline, the IT development community is simply not very good at estimation. We urge project managers to look for opportunities to develop and hone their estimation competencies and those of project team members. Such training might involve everything from a multi-hour workshop, to a multi-day short course, to broader courses as part of an advanced degree. All of these

28 McConnell, S., op. cit., 2006.

approaches are appropriate and can make an immediate difference. Once trained, project managers and their teams can help infuse the training and ideas into internal training programs or professional development initiatives. Ample opportunities for such training exist and project managers who continue to ignore them do so at their own peril.

Concluding Comments Insanity can be defined as doing the same

thing over and over again and expecting a different result. In effect, this is what the IT development community has been doing with respect to estimation for more than 20 years. The findings from Study 1 are remarkably consistent with similar findings first reported in the early 1990s, with “classic” estimation mistakes still being made—over-reliance on personal memory, use of intuition and guessing. Moreover, the common causes of poor estimation practices today have changed little over the same time span—change requests from users, users’ lack of understanding of requirements, poor problem definition and insufficient analysis prior to an estimate being made.

The results from our Study 2, which looked at project-level data, are slightly more encouraging. Our findings indicate that contemporary practices, including Agile development methods, seem to be making some headway toward improving project success. Interestingly, most projects met their stated requirements, produced usable products and services, and improved organizational value, leading to a high overall stakeholder satisfaction rating. However, somewhat paradoxically, most of the projects were late and many (40%) were over budget, reiterating the need for better time and cost estimation. Thus, today’s “typical” project can be characterized as a “successful failure.”

Many of the deficiencies in current projects can be traced back to poor estimation, so there is clearly room for improvement in estimation practices. We have provided specific guidelines for IT project managers to help them reverse the trend so that the estimation problems identified by our studies do not continue for another 20 years.

March 2014 (13:1) | MIS Quarterly Executive 29

IT Project Estimation: Contemporary Practices and Management Guidelines

Appendix: Survey Research Methodologies

Research Study 1: Organization-Level Analysis of Project Estimation

The survey questionnaire included 35 multi- part questions requiring responses on five-point Likert-type scales. Other questions required respondents to report the proportion of projects on which they carried out particular estimating practices. After piloting the survey with 40 IT professionals and revising it to improve clarity and completeness, the questionnaire was distributed in 2010 via an Internet-based survey tool to several hundred IT professionals affiliated with a major U.S. graduate school program on the Management of IT (alumni and current students; all working IT professionals).

The 60 respondents had a similar profile to those in Lederer and Prasad’s 1992 survey: they were responsible, educated and experienced IT professionals familiar with their current firms. While all respondents had a direct connection to project estimation, most (72%) were responsible for project/department management, with the remainder involved with estimation, systems analysis/design, programming and/or database administration. All respondents had a four-year college degree and at least some graduate-level education. On average, respondents supervised 31 employees and had 15 years of experience in IT with six years at their current companies.

The firms varied in size and industry. Their annual sales ranged from $900,000 to $25 billion with a mean of $4.5 billion. The average number of employees was 9,423 with a range of six to 77,000. On average, there were 1,539 employees in their IT departments, ranging from one to 18,000. IT department annual budgets ranged from $900,000 to $15 billion, with a mean of $1.1 billion. The primary industries represented were consulting/services (31%), banking/finance (24%), government (13%), education (7%) and healthcare (7%).

Study 2: Project-Level Analysis of Project Estimation

Study 1 provided a useful snapshot of overall organizational estimation practices and enabled us to directly benchmark the results against

the Lederer and Prasad 1992 study. However, we felt that a second study at a lower level of analysis—i.e., at the project level rather than the organizational level—would provide project managers with additional detail, richness and insights. Furthermore, by collecting data at the project level, we were more able to directly tie project estimation practices to project outcomes (schedule, cost, functionality), which Study 1 suggested continue to be a challenge for project managers.

In reviewing the results of Study 1 and talking with project managers, we realized that it would be much easier for them to talk about outcomes on a specific project (rather than projects in general). To add additional validity and reliability to our insights from Study 1, we therefore decided we needed to collect information on a wide variety and scale of individual projects. Armed with this project-level data, we felt we would be able to generalize our results for the broader population of contemporary IT projects.

In Study 2, we surveyed 115 IT professionals during 2011 and asked each respondent to provide data about his or her most recently completed IT project. Virtually all of the respondents indicated they were project/ program managers or directors, giving us confidence that they were in a position of responsibility over project timelines, costs and outcomes. Not surprisingly given the turnover in IT-related fields, most (86%) had worked for their current employer for less than 10 years and just over half (51%) had worked in the IT field for more than 10 years. A wide variety of industries were represented in the sample, with consulting/services having by far the highest number of respondents (see the table).

Additionally, the organizations in the Study 2 sample tended to be quite large. Annual revenue averaged $9 billion, and the average number of employees was 27,000.

As in Study 1, the projects reported on in Study 2 covered a wide spectrum of size, scope and functionality. This variance was expected because we asked each respondent to provide data on the last project that they worked on (not any project or a typical project). Projects ranged from a small web application development project costing several thousand dollars to a $350 million IT infrastructure project for a large international airport. In between, there were

IT Project Estimation

database and data center projects, business intelligence/analytics, ERP, CRM, systems

integration and many web-development projects. Project budgets averaged $11 million, with a median budget of $440,000.

About the Authors

R. Ryan Nelson R. Ryan Nelson ([email protected]) is Professor, IT Area Coordinator and Director of the Center for the Management of Information Technology (CMIT) at the McIntire School of Commerce of the University of Virginia. He received his Ph.D. in business administration from the University of Georgia in 1985 and spent five years on the faculty of the University of Houston before joining the University of Virginia. He has published several articles on the topic of project management in MIS Quarterly Executive and currently serves on that publication’s editorial board.

Michael G. Morris Michael Morris received his Ph.D. in Management Information Systems from Indiana University in 1996. He has previously served as a senior editor at MIS Quarterly and Information Systems Research. His research has been published in MIS Quarterly, Organizational Behavior and Human

Decision Processes, Personnel Psychology, IEEE Transactions on Engineering and Management and Decision Sciences, among others.

Survey 2 Primary Business Activity Industry Percentage

Consulting/services 50.67%

Government 10.67%

Financial services 6.67%

Healthcare 5.33%

Communication 6.67%

Education 4.00%

Retail 5.33%

Utilities 4.00%

Insurance 2.67%

Legal 2.67%

Manufacturing 1.33%

Copyright of MIS Quarterly Executive is the property of MIS Quarterly Executive and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.