Application 2 – Annotated Bibliography

UNDERSTANDING PROJECT FAILURE: USING COGNITIVE MAPPING IN AN INSURANCE PROJECT

Many projects fail, especially IT projects. The only way that companies can get better at performing projects is by learning from projects they have carried out. But tradi- tional practice of holding a lessons-learned session during or following a project may not allow organizations to examine the deep and "messy" reasons why projects fail, particularly with complex projects. Complex projects can best be understood by using modeling, such as cognitive map- ping. Cognitive mapping aids identification of causal chains and where these close in on themselves to form positive feedback loops, which helps to understand not only what went wrong, but also the reasons. Cognitive mapping is used to understand the impacts of management decisions on a project, both intended and unintentional. This approach is used here to examine a large software development project carried out by an insurance company which overran its original plan by several years. Some sug- gestions are made both for using such methods and lessons to be learned for future projects.

Keywords: lessons learned; complex projects; mapping

©2006 by the Project Management Institute

Vol. 37, No. 4, 55-71, ISSN 8756-9728/03

Introduction

T he research described in this paper has three distinct but interrelated

objectives:

• To explore methods by which learning in and from projects can take place:

What are the benefits of reviewing projects, and how can learning be

achieved?

• To learn about what went wrong or right in a major software development

project, Project X2; this was to be used a vehicle to “learn how to learn”

(Cooper, Lyneis, & Bryant, 2002).

• To make recommendations for the organization's future projects to improve

both project execution and results and also learning between projects.

History of Project X

In late 1997 an organization, called here AutoInsurance Ltd (AIL), recognized that

the core software package that supported the business (called InsSoft in this

paper) would shortly begin to place limits on the organization's development.

InsSoft was also not year 2000 compliant. Tenders were invited for the provision

of a replacement package; the contract was won by a supplier, and Project X initi-

ated. By early 1999 it was decided that the project was not going to succeed and

the contract was terminated. Subsequent lawsuits by AIL and counter lawsuits were

eventually settled out of court.

AIL then reissued the invitation to tender. This second contract was won in

June 2000 by the supplier of InsSoft, a software systems supply company we will

call SSS. The new project was called Project X2 and was split into four phases:

• Integrating an industry-standard quotation engine into InsSoft

• Establishing a model office environment at AIL

• Developing, implementing, and integrating SSS's customer relationship

management (CRM) package

• Replacing InsSoft with new software, based upon SSS's proprietary packages,

and integration.

Every stage of the process finished later than planned. The delivery of the solu-

tion architecture was late. The expected delivery date for the Release C design that

was later than the original planned delivery date; this date too was missed. After

Release C was implemented further delays meant that the planned Release D was

STEPHEN ROBERTSON, Glasgow

TERRY WILLIAMS, Southampton University

ABSTRACT

55S E P T E M B E R 2 0 0 6 P r o j e c t M a n ag e m e n t J o u r n a l

56 S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

split into Releases D1 and D2. Release D1 was delivered and

implemented, but due to further delays Release D2 was fur-

ther split into Releases D2.1 and D2.2. Release D2.1 was

achieved on time, but with less functionality than had been

planned, so the final release was again split, into releases

D2.2A, B, and C. Release D2.2C was delivered to AIL for user

acceptance testing (UAT) in June 2003, with a plan to start

implementing the software at the end of the year. Delays to

the testing process resulted in another delay, and the soft-

ware implementation eventually began in June 2004. This

was planned to take at least six months to fully implement

across all functions in AIL. In order to achieve this goal,

some compromise was made on the quality standards

agreed for the completion of UAT, which resulted in over

800 known defects outstanding when implementation started.

Despite the delays and defects (and therefore

workarounds), there was a sense at AIL that the project had

been a success. AIL's managing director indicated that he

was very satisfied with the end result. Business benefits were

immediately identified in some areas and, with familiarity

and as defects are resolved, there was a good expectation

that this would be improved.

Literature

Roots of Failure in Projects

Despite a belief that project management techniques have

matured, the rate of failure of projects has never been high-

er (Cooke-Davies & Arzymanow, 2003). There are various

surveys on project failure; all agreed that failure is not

uncommon. Reichelt and Lyneis (1999) reported the find-

ings of various other authors, that projects which overrun

are more common than projects which complete within

original time scales, overruns likely to be between 40% and

200%; that fewer than half of projects examined in one sur-

vey met cost and schedule targets; and that only one third of

World Bank projects met their aims, with typical delays of

50%. Jørgensen and Sjøberg (2004) reported another survey

showing only 17% of projects meeting all three aspects of

the project triangle (cost, time, and scope, Gray, 2001), with

typical cost overruns as high as 189%. Worse still are IT proj-

ects where it is reported just 3% are considered to be a suc-

cess (Journal of Management Development, 2001). Keil and

Robey (1999) reported a survey that found that half of all IS

projects in the U.S. in 1995 failed, costing a total of

$140 billion. What constitutes project success or failure? Most fre-

quently a project is considered a failure if it fails to meet its targeted cost, time, or scope. Commonly a fourth criterion is also included, the benefits accruing to the organization as a result of the project. White & Fortune (2002) found seven criteria used for judging the success of a project, all related to the above four. Shenhar, Dvir, Levy, & Maltz (2001) describe four dimensions of success, with 13 measures per- taining thereto, which are similar:

• Project efficiency (meeting schedule and budget goals, or, “meeting design goals” Dvir, Raz, & Shenhar, 2003)

• Impact on the customer (meeting functional perform- ance and technical specifications, fulfilling customer

needs, solving a customer's problem, customer using the product, customer satisfaction)

• Business success (commercial success and creating a

new market share)

• Preparing for the future (creating a new market, prod-

uct line, or technology).

The literature suggests a great many causes of failure of

projects. In his study of 136 European projects, Cooke-

Davies (2002) found 12 “real success factors” grouped under

three headings:

• Factors critical to project management success: partic-

ularly relating to project risk management (PRM)

(PRM education, ownership of risks, a maintained

risk register, and a PRM plan) but also documented

organizational responsibilities, project stage dura-

tions below three years, a mature scope change man-

agement process, and maintenance of the

performance measurement baseline

• Factors critical to success on an individual project-

namely, cooperation between project managers and

operational managers within the business

• Factors critical to consistently successful projects-pro-

gram management able to resource projects matching

business strategy, metrics linking project performance

with anticipated future success, and (interestingly) an

effective means of learning.

Cooke-Davies does not make explicit the role of people

in delivering these, but does say “people perform every

process, and it is the people who ultimately determine the

adequacy. Thus the 'people' side of the success factors is

woven into their very fabric.”

Moynihan (2002) identified a number of “people-prob-

lems” against which he advises the contractor (in software

development projects) to protect itself: unrealistic customer

expectations, lack of ownership of the project by anyone in

the client organization, disagreement about the project's

goals within the client organization, lack of skill of the

client's project manager, reluctance of users, and politics

within the client organization. The first of these is picked up

by Jørgensen and Sjøberg (2004), who found that unrealis-

tic customer expectations can lead to significant underesti-

mation of the project's duration, particularly where the

estimate provided forms the basis of a bid for a software

contract. Similarly, Jiang, Klein, Chen, and Lin (2002) found

an overlapping set of people factors affecting success,

including user involvement and participation and executive

management support.

The culture of the organization undertaking the project

is a crucial element in these factors. Gray (2001) found that

in many organizations there was, to some degree, a feeling

of threat against the project's participants should the project

fail (typically to their career prospects, reputation, financial

returns, and self-image). There appeared to be an acceptance

that this was a necessary evil, if the project were to be a suc-

cess. However, it was found that where such behavior was

57S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

common, there was a greater propensity for projects to fail.

Furthermore, where the organization's culture favored “vol-

untarism” (i.e., free expression, questioning of management

decisions, participation in defining goals, intrinsic work sat-

isfaction) the likelihood of success was increased.

Participants from the 15th IPMA World Congress pro-

posed eight factors leading to project failure (Journal of

Management Development, 2001). One was the “Wrong

Leader.” Five types were identified: two of whom damage

the project-the “overt saboteur” and the “passive resister”;

two of whom do little to benefit it-the “non-committed”

and the “well-wisher”; and only one beneficial-the “fully

committed.” A second was “Ineffective Controls”: financial

controls concentrate on cost rather than time, so delays

caused in an effort to save money can end up costing much

more than the planned saving; project management tends to

look backward comparing progress to-date against plan, and

spend too little time looking forward, anticipating issues

and making timely corrections. Others included inappropri-

ate team and poor communication.

Dvir et al. (2003) found that, although there was no

doubt that some basic level of planning was required, there

was no correlation between planning and success. However,

they did find a significant correlation between success and

the development of requirements for the project, and rec-

ommended that the project's goals and deliverables should

be defined (a view echoed by Jiang et al., 2002).

The Need for Learning from Projects

If organizations are to get better at complex projects, they

need to learn from project experience. They must examine

the events of the project, and also the behaviors that may

have led to failure (or success): what went wrong? How did

that impact the project at hand? What was the management

response at the time? What impact did that response have?

How could such events be avoided in the future? As Disterer

(2002) noted, most firms fail to review projects and as a

result tend to repeat bad decisions and errors. According to

Ayas (1996), “The ability to sustain success and significant

improvement in projects over a long period of time depends

mainly on the capability to learn from experience.”

Schindler and Eppler (2003) went further and suggested

that an organization that is able to learn from project expe-

riences is one that is able to make comparisons between the

various projects it carries out and record the best problem-

solving practices it finds; and it will be able to minimize risk

in future projects by knowing potential problems and taking

appropriate action much earlier. The effect of this is an

organization able to “develop project competencies that

lead to a sustainable competitive advantage” (also see simi-

larly Barker & Neailey, 1999).

Project management methodologies have recognized

the need for learning from the past. PRINCE 2 promotes the

use of a “lessons learned log,” which is established early in

the life of a project and maintained throughout (Office of

Government Commerce, 2002). This provides the basis for

writing a formal lessons learned report at the end of the

project (to “ensure that all lessons learned during the proj-

ect are annotated for the benefit of future projects” and

“report on whether the project management activity itself

has been a success or not”) (Central Computer and

Telecommunications Agency, 2001). Similarly, the need to

take note of lessons learned is recognized in A Guide to the

Project Management Body of Knowledge, which emphasizes the

need to record not only the events that occur within and to

the project, but also the management actions taken as a

result and the reasons for those actions (Project

Management Institute, 2000).

However, there is a considerable body of evidence to

suggest that, despite the self-evident benefits of learning

from past successes or failures, in practice this is seldom

done (e.g., Cooper et al., 2002; Disterer, 2002; Garvin, 1993;

Schindler & Eppler, 2003; Williams, 2003a; 2004). There are

a number of potential reasons.

First, there is a lack of understanding about the validity

of learning from projects, so the activity may be seen as a

waste of time. There is a tendency for managers to consider

that one project's experiences are unique so offer no benefit

to future projects-or there is so little in common that “sepa-

rating the differences from the similarities would be difficult

if not impossible” (Cooper et al., 2002). Williams (2003a

and references therein) noted a similar effect whereby man-

agers “avoid the discipline of drawing in experience from the

[project] before” and that “the organization that thinks it

has a unique situation is unlikely to gain from the experi-

ences of past works or others. True systemic causes and

transferable project management lessons are there to

be learnt.”

Second, previous bad experiences, where no benefit has

been seen to be gained from this activity, may prevent future

attempts from being made (Williams, 2003a; 2004). This is

especially true if the task is allowed to degenerate into a

mere paper exercise, where a formulaic approach yields triv-

ial and poorly thought through results, which are not seen

to lead to any beneficial change in future projects.

Under these circumstances a third factor is likely to be

strong, where time pressure is experienced, ruling out bring-

ing the project group together for any kind of learning expe-

rience (Schindler & Eppler, 2003; Williams, 2003a; 2004).

Usually after the end of a project the company has a need to

ensure that the personnel involved are deployed onto a new

project as soon as possible. Little or no time is given to post-

project review (Cooper et al., 2002), especially when the

project has been a failure. There tends to be an unwilling-

ness to spend time dwelling on past failures (see also

Sense, 2004).

Using time pressure as a reason for not conducting a

post-project review may sometimes be a convenient excuse,

when in fact the fourth reason may be the (unspoken) fear

that an examination of the project's failures will elicit evi-

dence of personal failures. People at all levels in projects are

often unwilling to admit to their own mistakes (Abdel-

Hamid & Madnick, 1990) and the impact these may have

had on the course of a project. This is especially true when

58 S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

there is a sense that negative sanctions will be used should

errors be uncovered (Schindler & Eppler, 2003; Smith, Keil,

& Depledge, 2001). Even if reviews do take place, the lack of

frank and honest self-appraisal by individuals involved

means that it may not be able to achieve all that it could.

Finally, the difficulty in determining the true cause of

failure (or success) means that it is often difficult to make

any review more than a banal recital of the obvious. Cooper

et al. (2002) asserted that even successful managers find it

hard to identify the best practices. This is particularly true

because projects are becoming more complex (Baccarini

1996; Helbrough, 1995b; Jaafari, 2003; Williams, 1999a ).

Complexity is caused from two elements. First, a complex

project consists “of many varied and interrelated parts …

operationalized in terms of differentiation and interdepend-

ency” (Baccarini, 1996); for such, traditional project man-

agement decomposition-type methods are inadequate as the

interaction between the parts of the project means that the

project is more than the sum of its parts (Williams, 2002;

1999a). Baccarini discussed his definition of project com-

plexity in terms of two dimensions: organizational and tech-

nological complexity, related to the structure of the

organization and the project. The second element is uncer-

tainty; although there is some debate about the role of

uncertainty in project complexity, the inclusion of uncer-

tainty allows complexity to be defined in a more over-arch-

ing sense (Williams, 2002). Uncertainty itself has two

subdimensions (Turner & Cochrane, 1993): goal uncertainty,

typical of software development projects where the require-

ments for the product are poorly understood or defined;

and methods uncertainty, so standard project management

plans cannot be created because the constituent tasks and

dependencies are not clearly understood.

The Nature of Learning

Learning has been defined as “the process by which knowl-

edge is created from experience and the path by which

improvement takes place” (Kotnour, 2000). Learning is dif-

ferent from experience, as not everyone learns from experi-

ence; there is an activity involved in learning; a process that

an individual has to initiate. A learning organization has

been defined as “an organization continually expanding its

capacity to create its future” (Kotnour, 2000), and as “an

organization skilled at creating, acquiring, and transferring

knowledge, and at modifying its behavior to reflect new

knowledge and insights” (Garvin, 1993). It is insufficient

simply to generate new knowledge-in the form of docu-

ments, processes, and operational procedures-unless that

knowledge is embedded in new ways of working for the

organization (Williams, 2003a). According to Ayas (1996),

most of the knowledge in an organization is tacit, existing

only in the minds of its people, however it can be made

explicit by sharing it with the surrounding workgroup, and

the behavior of the organization can change as a result of

the new knowledge.

Mechanisms of Learning from Projects

Many project management methodologies emphasize estab-

lishing lessons learned both during and at the end of proj-

ects. According to Kotnour (2000), during the project,

lessons learned benefit the project itself; at the end of the

project (although this may also happen during the project)

the focus moves on to lessons that are transferable to other

projects. He describes how allowing, and encouraging, the

project team to spend time reflecting on the actions taken

during the project, and the good or bad outcomes of those

actions helps to break down barriers to learning within the

organization. This is supported by documenting the lessons

learned, including which actions to take and which to avoid.

Learning is fed back either to the project in hand or to sub-

sequent projects according to the “plan-do-study-act” cycle.

Kotnour describes four dimensions of the lessons

learned process:

• When: Lessons can be learned throughout the project

as opportunity arises, throughout the project at regu-

lar meetings, or at the end of the project.

• What about: Lessons should be learned about tasks

with minor and major or no problems.

• How: Issues can be identified by comparing actual to

planned performance, and by remembering.

• What: Included in a lessons learned could be the orig-

inal plan, the actual results, the problems encoun-

tered and the tasks that went well.

Barker and Neailey (1999) suggested that the failure of

post-project reviews has a number of causes, a key one being

the failure to provide a context for the review. They suggest-

ed that a useful context is innovation, which they took to

be a combination of “conception” “invention,” and

“exploitation.” They recommended a four-stage approach

based upon their experience of a project within the

automobile industry:

• Individual learning logs to encourage team members

to reflect upon and record their own learning. The

structure of the log helped individuals achieve “more

detailed and specific recollection of learning,” and the

logs supported the kind of thinking that could lead to

innovation.

• Functional learning reviews, bringing together indi-

viduals from within a functional grouping to assist

them to combine their learning and identify learning

for that section.

• Whole-team learning review: The leaders of each func-

tion are brought together in a workshop to review the

output from others' learning reviews and jointly

develop plans for change.

• Communication of learning: The importance of doc-

umentation is recognized, but greater emphasis is

placed upon informal, verbal communication.

A major failing of such approaches is that while they

give excellent suggestions on how the tacit and implicit

knowledge generated through the life of a project is made

59S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

exist, it is interesting to note the effect management actions

have in that loop, as often interventions designed to help

can in fact make matters worse (Eden, Williams,

Ackermann, & Howick, 2000; Howick & Eden, 2001). Once

several maps are generated they can be combined into one

map, including all of the information available from the

interviews and workshops.

Complementary to cognitive mapping, which can pro-

vide a qualitative analysis of projects, is system dynamics

(SD), which can provide a quantitative analysis (Cooper,

1994; Cooper et al., 2002; Rodrigues & Bowers, 1996;

Williams, 2002). For example, it can help to quantify the

impact that clients have on a project's outcomes, even when

the assessment might involve subjective aspects. Rodrigues

and Williams (1998) reported a typical scenario where an

early milestone is about to be missed. The client sees this as

an indication of failure, and cannot accept this. The con-

tractor's only response may be to recover the short-term

milestone, only to incur worse overruns later in the project.

In cases such as this, SD may be of benefit in helping to

quantify the impacts of the actions, and in providing a

rationale that is otherwise missing. In addition to describing

a project post-mortem, SD can help the project manager to

make decisions for the future (Rodrigues & Williams, 1997).

For example, where a change to a project's scope is threat-

ening to increase its time scales, the traditional approach to

project management is likely to lead to the introduction of

additional resources; however, adding resources to an

already-late project may in some cases lead to no significant

benefit, or worse, to further delays. This is the basis for both

Brooks' law and Cooper's $2,000 hour (Cooper, 1994). In

such circumstances, SD may help to determine the appro-

priate balance, which minimizes delay (Williams, 1999b).

According to Lyneis, Cooper, and Els (2001) the use of SD

tends to be around the strategic and/or tactical management

of projects, rather than operational.

Implications for This Study

This literature had a number of implications for the study

in AIL:

• Before an approach to learning from Project X2 could

be decided, it was essential to establish whether it was

a complex project or merely big. If the former, a map-

ping approach looking for positive feedback dynam-

ics might be valuable; if the latter, then a more

straightforward lessons learned process may be as

effective, and more time-efficient.

• Assuming that the project was complex, interviews

carried out with key project personnel using cognitive

mapping might be the most effective way to elicit the

information required for the analysis, making it more

likely to avoid pressure to conform to any one point

of view.

• Analysis should focus on management interventions

designed to bring the errant project back on course,

and consideration given to all of the consequences,

intended or otherwise.

explicit and documented to enable effective sharing within

an organization, they have little to say about how new

knowledge can be created (Williams, 2003a). Projects are

becoming increasingly complex, and outcomes are often dif-

ficult to relate to the actions and events that shaped them.

In such circumstances collecting data on what was planned,

and what actually happened and ever more-detailed recol-

lections about the “facts” of the history of the project is rel-

atively straightforward and time consuming; however,

understanding how all of these things are interrelated and

interacting is much less trivial (Williams, 2002). Schindler

and Eppler (2003) noted that while traditional approaches

are good at answering questions such as “what,” “where,”

and “how many”; they are not good at getting at the more

difficult questions of “why” and “how”: questions which are

much better answered by reference to “stories” (Schindler &

Eppler, 2003; Williams, 2002).

As we have discussed, traditional project management

decomposition-type techniques are inadequate to explain

the behavior of complex projects (Williams, 2002; 2004).

One event or issue may have an adverse impact upon many

activities, and may create new relationships between activi-

ties; this becomes even more complicated where there are

multiple such issues; the impact upon an activity may not be

direct or obvious (e.g., impacts on the project team's

morale); there is little or no recognition of the management

inputs; often impacts to “noncritical” tasks can often have

significant delaying effects; and finally the eventual delay to

a project may not be simply the duration of the original

delay. For example, AIL started a piece of work requiring a

deliverable from the supplier by its mid-point, or else it

would have to be abandoned and restarted from the begin-

ning; the deliverable was late, and the early work wasted.

The problem lies in the nature of complex projects and in

their inherent “messiness”: “… explanations center around

the systemic interrelated set of causal effects; the dynamics

set up by these effects including feedback loops; important-

ly, the management response to project perturbations;

and the resulting exacerbation of the feedback loops”

(Williams, 2002).

One answer to this complexity is to employ modeling

techniques, , in particular, cognitive or causal mapping

(Williams, 2000; 2002; 2003a; 2004), which is specifically

designed to help tackle “messy” problems (Eden, 1988). In

this approach, interviews are carried out with people who

have been involved in the project, either one-to-one or as

part of a group workshop. The circumstances and events of

the project are discussed and explanations sought for them.

Concepts that describe events, issues, management actions,

etc. are noted, and any causal link between them is drawn

in, i.e., where one concept led to another, or led to its

increase, the two are linked by cause and effect. Where the

effect is to decrease, a negative sign is assigned to the link.

Each map is examined and feedback loops are identified.

Where there are no negative signs in a loop, or the total

number of negative signs is even, the loop is a positive feed-

back loop, or “vicious circle.” Where such feedback loops

60 S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

• Particular attention should be given to looking for

signs of positive feedback dynamics.

Methodology

This research involved the study of a large software develop-

ment project (Project X2) undertaken by AIL (the client)

and SSS (the contractor). A careful study using cognitive

mapping was carried out to determine the causes of project

time overrun. Particular attention was given to the manage-

ment actions taken to resolve the developing overrun, to

establish whether secondary effects of these actions may

have contributed to the end result. Following this analysis,

some general recommendations were made to AIL's man-

agement team in order that future projects may avoid, or at

least minimize, such overruns.

Clearly, this study tries to extrapolate general learning

from one specific example, so is phenomenological.

Phenomenological research is effective at making explicit

perceptions and interpretations of individuals and in chal-

lenging assumptions, and is generally seen as being descrip-

tive rather than providing explanation (Lester, 1999). Care is

therefore taken in describing any generalized findings to

explain how these were arrived at and to put them into the

context of other published findings. Easterby-Smith, Thorpe,

and Lowe (2002) put most research in the social sciences on

a two-dimensional plane-one dimension being character-

ized by the positivist/social constructionist poles and the

second by the level of involvement of the researcher, from

“involved” to “detached.” Social constructionism is charac-

terized by the assumption that “reality is not objective and

exterior, but is socially constructed and given meaning by

people.” The focus of such research is on gaining an under-

standing of how people perceive and understanding the

phenomenon under study, and how this is affected by the

paradigms they hold. Although this research lies quite firm-

ly on the social constructionist side, it does recognize the

objective reality of the situations under study. Regarding the

second dimension, in this research the researcher was an

integral part of the AIL management team who undertook

Project X2; therefore, it was impossible for the researcher to

abstract himself completely from the process, and it will be

noted throughout that this does limit the researcher's objectivity.

The approach taken followed Williams (2004).

Cognitive mapping (Eden, 1998) was employed to interview

a number of key AIL managers, who had responsibility for

aspects of Project X2 in looking to gain an understanding of

the important events within the project that led to its signif-

icantly late delivery. Interviews rather than a group work-

shop were chosen for a number of reasons (Eden &

Ackermann, 1998):

• The participants came from various levels within the

organization, and there was likely to be some dis-

agreement among them as to the impact of some

management decisions; this might cause material not

to be surfaced in a group context for fear of negative

impacts for more junior members.

• This gives interviewees more time to think, giving a

richer, deeper understanding of the subject matter.

• It avoids the risk that the louder, more forceful per-

sonalities are the only ones whose opinions are heard.

• It avoids some of the potential hazards of working in

groups, e.g., “social loafing” or “group think.”

• It is less time-consuming for each of the participants.

There are, however, a few drawbacks of interviews rela-

tive to workshops. In particular, the process is more time-

consuming for the facilitator, with several interviews and

subsequent analysis of the resultant cause-maps, and the

combination of the maps into one. Also if there is no sub-

sequent group workshop the interviewees can feel detached

from the end result and therefore not accept the

output fully.

An SD approach, although generally useful in analysis

of this type, was not adopted. This was because the exercise

was being carried out from the perspective of, and on behalf

of, the client and much of the hard, numerical data required

to support an SD approach (e.g., cost profile) is only avail-

able to the supplier, who would be unwilling to disclose this

information. Under these circumstances an SD approach

was not feasible, and so the approach adopted relied on cog-

nitive mapping only.

Findings and Analysis

Was Project X2 Complex?

Complexity is not a factor of size alone, but rather arises from the interaction and interdependence of many subele- ments within the project, within the organization(s) engaged in the project (organizational complexity), within the project itself (technological complexity), within the product, or due to uncertainty within the project. So did Project X2 display any of these factors, and is it complex? First, organizational complexity. The primary parties involved in the project were AIL and SSS, but there were a number of other parties involved:

• Part of the system to be delivered included computer telephony integration (CTI). The development of this software functionality was subcontracted by SSS to a U.S.-based company, and the development was car- ried out mostly in the U.S. As the client, AIL had little contact with this company.

• The system was made up of three major software packages tightly integrated. One was owned by anoth- er U.S.-based company with whom SSS had a sole dis- tribution rights deal, but SSS did not own or control the code. During the project, SSS bought the product's intellectual property.

• A key feature of the product being developed in Project X2 was a subsystem that calculates a premium for private motor car insurance based upon various risk factors using a quotes engine. At the start of Project X2, SSS owned the quotes engine and devel- oped it. During the project, a management buy-out occurred. The part of SSS covering the quotes engine was bought and established as a new company.

61S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

• On the client side, a number of other organizations

were involved in the project. At first, project manage-

ment and test management were provided by external

consultants. However, AIL did not feel that they were

meeting expectations and terminated the contract. A

consultant from another consultancy was hired as

project manager. Another external consultant was

hired as test manager, and more consultancy support

was used toward the end of the testing effort.

Therefore, although the number of parties in the project

changed over time, there were never less than five different

commercial organizations involved. The coordination of

effort between these was challenging at times. For example,

when the test manager first became involved, the legal teams

of SSS and his home company meant it was difficult to agree

on the wording of a nondisclosure agreement, which meant

that he was not allowed to see any documentation or attend

meetings with SSS for more than two weeks.

Second, technological complexity. The nature of the

project itself was complex, and increased in complexity over

its life. Having several release dates timed relatively close to

each other meant that releases were at different states of

maturity at the same time, all requiring input from the same

resources in both SSS and AIL. For example, while release C

was being implemented, release D1 was in the user accept-

ance test phase, and release D2 was in the detailed specifi-

cation phase.

Third, the product was also very complex, as it is made

up of several parts. The major components of the final solu-

tion were a CRM tool, an insurance sales and administration

quotation application, and an accounts package. The major

components themselves consisted of several interdependent

parts. The CRM tool was integrated to several other compo-

nents: the call center's preexisting telephone automated call

distribution switch, which directs inbound telephone calls,

and the call center's automated dialer, which makes out-

bound customer calls; a bespoke workflow system that

enables work to be recorded and stored for later or trans-

ferred around the organization; postcode lookup software;

and AIL's unique specialist database. The quotation product

too was complex and required several component parts to

perform many interdependent functions-namely, to provide

new business quotations and allow for their confirmation

(requiring integration to an external vehicle database, held

by the government licensing authority DVLA); to provide

quotations for permanent or temporary changes to the risk,

and allow for their confirmation; to produce customer doc-

umentation including legal documents (requiring integra-

tion to production and queuing systems); and to

communicate new policies to the underwriting insurer,

requiring integration to an electronic data interchange sub-

system. Also, the products were integrated with a credit-card

verification and payment system and a direct banking com-

ponent for the establishment and maintenance of

direct debits.

This complex functionality depended on a highly com-

plex technical infrastructure. The application was “fat

client,” which means that most of the business logic resided

on the user's desktop PC. However some business logic

along with workflow and external components resided on a

middle tier server. The telephony was specialized. One

major database ran on Oracle. There are other SQL databas-

es that ran under Windows. The database servers all ran with

high availability clustering, while the middle tier server was

split into a set of seven load balancing servers. Each of these

elements required specialist knowledge to maintain

and develop.

In conclusion, Project X2 was complex. It met

Baccarini's (1996) definition. Organizationally and techni-

cally, it consisted of many subelements that were highly

interdependent. The product also consisted of many inter-

dependent software and hardware components that were

interrelated and interdependent. As such, standard lessons-

learned type approaches were likely to be inadequate for its

study. Instead, as previously discussed, a cognitive mapping

approach was more likely to help achieve meaningful learning.

Identification of Candidates for Interview and Construction

of Cognitive Maps

The AIL sponsor for Project X2 was the information services director, and reporting to her was a project manager, an external contractor. Interviews were carried out with the project manager and cognitive maps drawn. Cognitive maps were also drawn for the business process manager (BPM), who reported to the sponsor and was responsible for ensur- ing the system met the business' needs and delivered bene- fits; it should be noted that the BPM is the main author of this paper, and although every attempt has been made to remain objective, this may not always be possible. Two cog- nitive maps were produced based upon the BPM's experi- ence: the first showedevents, decisions, and situations that occurred leading up to the launch of the application; the second showed the situation following launch.

Pre-Launch Map

Figure 1 shows the resulting map. The concepts were color- coded into four categories; in this paper, ovals with bold writing represent the main outcomes, other ovals represent events external to AIL, rectangles represent AIL management decisions, and rounded rectangles represent other concepts. Obviously, color-coding is extremely useful in drawing up and analyzing these maps, which does not come out fully in gray scale. As previously discussed, one of the most signifi- cant structures to find in such maps are positive and nega- tive feedback loops, where a chain of cause and effect loops round and joins up again at some earlier point; all the arrows being positive (or an even number of negative arrows) increasingly leads to more and more of whatever is being described, good or bad. Analysis of the map using Decision Explorer (Eden et al., 1992) shows that there are a great many loops (1,633), although many of these are slight variations on the same theme. There are however a number of significant loops that can be considered.

62 S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

Destructive Positive Feedback Loops

One significant loop is shown in Figure 2. Ongoing effort to

complete the design resulted in the final design for the

application not being agreed, meaning that the supplier's

developers had insufficient work to keep them fully occu-

pied. The supplier, driven by the need to keep costs down on

the fixed-price contract (concept 4), chose to put their devel-

opers to work writing code in the hope that there would be

little change required when agreement was reached. The

supplier subsequently had to concede that some work had

begun on design that needed to be changed (concept 18).

This resulted in changes being required in pieces of func-

tionality already written and rework of development effort.

To make matters worse, other interdependent subelements

often had to be redesigned in order to accommodate these

changes, which required more discussion and extra design

effort, so the cycle begins again. One thing not clear from

the maps is that this cycle persisted for over a year. This loop

is related to a similar loop (loop 2 in Figure 2) in which

management decisions taken by AIL aggravated loop 1.

Through negotiation with the supplier, AIL not only agreed

to, but also encouraged the parallel working that led to the

positive feedback loop being established.

Clearly these decisions by SSS (concept 7) and AIL

(concept 12) were intended to mitigate the delays to the

project, and although they did manage to ensure that all

resources were fully utilized, they also meant that resources

were not necessarily being used on tasks that were con-

tributing to the completion of the project. The resulting

loops caused the delivery date to be revised many times.

Without access to SSS internal documentation it is hard to

be precise about the degree, but they added at least one year.

Another significant loop can be seen in Figure 3, which

shows another impact of parallel activity in design and

development. When testing was performed, a large number

of apparent defects were discovered. Many of these were

genuine defects, a significant proportion of which were

attributed to design documentation, which was incomplete

when the code was developed. The remainder were misun-

derstandings on the part of the tester who recorded the

defect, again due to incomplete design documents. The time

taken by test and design resources to analyze these defects

had an immediate impact upon the time scales of the proj-

ect, whether they were ultimately determined to be a gen-

uine defect or not. Another impact that can be seen in loop

4 (also in Figure 3) was where genuine defects were discov-

41

Introduce

joint-working

38

Changes in

the design

17

Client makes

compromise

on design

34 Disagreement

whether issue is a defect or is

working as designed

31 Many (apparent) defects found in

software delivered

33 Time to analyze

failure

32 Time to perform

testing 30

Software accepted for testing with

many known defects

29 Software delivered

with many known defects

37 Business continues

to change

26 Project costs

5 Supplier need to minimize waste

22 Supplier costs outweighing

income

6 Project late

4 Fixed-price

contract

43 Supplier under-

estimate of project duration

42 Supplier under- estimation of

project complexity

1 Supplier late

in delivering design documents

2 Supplier costs

outweighing income

8 No agreed design

44 Encourage CSC and KFI staff

to work overtime/ produce more

7 Increase parallel

working between design and development

3 Software development continues based upon

incomplete design

35 Many agreed

defects 23

Rework of completed

development

36 Many issues accepted

as “working as designed”

24 Rework of

interrelated elements

19 Changes required

to functionality already developed

12 Encourage parallel

working to maintain schedule

18 Supplier makes

compromise on design

20 Continued effort

to complete design

10 Many comments by

client on design documents

11 Contractual

negotiations on differences

15 Lawyers consulted

9 Many errors and omissions

in design documents

13 Senior

manager involvement

28 Supplier senior

manager time expended

14 Client senior

manager time expended

21 Discussion to

determine correct design

25 Scope of product

40 De-scope: Accept

more defects for go-live

39 De-scope: Remove

data migration

27 Need to

avoid delay

16

Agreements

on elements

of design

Figure 1: BPM’s pre-launch map

63S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

ered. This resulted in rework of the

affected components and often of one

or more interrelated components. This

was aggravated by the extended time

taken to analyze the defect before it

was assigned to a developer for a fix to

be produced.

It should be noted that, although

it is not apparent from the map, these

two sets of loops (1 & 2 and 3 & 4) did

not occur over the same time frame,

although they did overlap somewhat.

The loops 3 and 4 relating to testing, in

general started as loops 1 and 2 relat-

ing to the original design work were

coming to an end.

Given that time scales for this

project were extending significantly, it

is unsurprising that the business did

not stand still during this time. This

7 Increase parallel working between

design and development

3 Software development continues based upon

incomplete design

23 Rework of completed

development

6

634

5

24 Rework of

interrelated elements

21 Discussion to

determine correct design

19 Changes required

to functionality already developed

12 Encourage parallel

working to maintain schedule

2 Developers without useful work to do

8 No agreed design

20 Continued effort

to complete design

11 Contractual

negotiations on differences

27 Need to

avoid delay

10 1

18

33 38 31 35

44 41 40 39

16 15 13

Figure 2: Partial map showing loops 1 & 2

resulted in AIL management decisions

to request changes to the design to

accommodate the changing business,

which resulted in a further positive

feedback loop (loop 5; Figure 4) being

established. The effect of changes

being requested was to cause further

rework of already completed tasks, fur-

ther delaying the project allowing the

business to experience more change.

As discussed there were a number

of management interventions in this

project, most of which aimed to limit

the delay. However, some, counter-

intuitively, had the opposite effect.

One significant decision was to

encourage the use of overtime in both

AIL and SSS. Although overtime can be

an effective means in resolving a short-

term issue, over the long-term there

tends to be a law of diminishing

returns, after some time the units of

work per hour decreases, meaning no

more work is achieved. Worse, the

quality of the work being achieved is

adversely impacted (Cooper, 1994),

leading to two further loops (6 and 15)

shown in Figures 5 and 6. In Project

X2, this combined with the decision to

accept the application when it was

delivered-despite the fact that it had a

large number of known defects. These

loops (Figure 6) resulted in AIL's test-

ing time being extended, and led to

further rework (by SSS) and subse-

quent retest (by AIL) further delaying

the project.

Beneficial Negative Feedback Loops

Not all AIL management decisions

caused positive feedback. As time went

on and the project became very late,

some decisions were taken to resolve

issues extending the testing time scales.

One was to move some SSS resource

from their base to AIL's site, to enable

more joint working (the “one-team

approach,” concept 41). This resulted

in a negative feedback loop in which

the time taken to analyze apparent

defects was decreased allowing

resources to concentrate on the real

testing effort (loop 7 in Figure 7) and

also meaning the development team

learned of defects earlier, reducing

rework (loop 8 in Figure 7).

7 Increase parallel working between

design and development

3 Software development continues based upon

incomplete design

23 Rework of completed

development

24 Rework of

interrelated elements

21 Discussion to

determine correct design

20 Continued effort

to complete design

34 Disagreement

whether issue is a defect or is

working as designed

35 Many agreed

defects

31 Many (apparent) defects found in

software delivered

32 Time to perform

testing

26 Project costs

36 33

33 30

37 1

8

30

33 38 19

1 22

19

14

5 Supplier need to minimize waste

2 Supplier costs

outweighing income

6 Project late 4

25 43

10 1

8

19

Figure 3: Partial map showing loops 3 & 4

64 S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

Two further significant decisions

were taken: one was to change the

established criteria for exiting testing

to allow more defects (concept 40), the

other to de-scope the project, tem-

porarily removing the data migration

task (concept 39). These two decisions

arising out of a desire to minimize

delay in launching the application to

the call center created a further two

negative feedback loops that helped

reduce the overall time scales (loops 9

and 10 in Figure 7). However, not all of

the impact of these decisions were ben-

eficial, as the resulting de-scoping of

the product had some negative

impacts on the business as will be

seen later.

Trigger Events

A useful analytical tool in Decision

Explorer is to examine the “tails” of the

map, those concepts that only have

arrows coming out. These should be

trigger events, which initiate the loops

in the first place (otherwise, it would

indicate that there was more material

to be elicited). In this map there are

two trigger events. One is that the orig-

inal contract for the project was on a

fixed-price basis (concept 4). This

meant that from the start the supplier

was very determined to keep costs at a

minimum, and never to have resources

underutilized. The other trigger was

that the supplier initially underesti-

mated the scale and complexity of the

work to produce this software solu-

tion. It seems likely that the underesti-

mation was at least in part caused by

the supplier's failure to separate the

“price-to-win” bid from a realistic esti-

mation of the effort required, and that

they suffered from over-optimism in

making their estimates (typical in such

situations-Jørgensen & Sjøberg, 2004).

Failure to fully grasp the complexity

and underestimating the time required

led to the initial project overruns,

which forced the management correc-

tions that in turn started the positive

feedback loops. It also resulted in the

errors and omissions that contributed

to the failure to reach agreement on

the design.

23 Rework of completed

development

33 35 19

26 1

25 43 32 10 1

8 19

24 Rework of

interrelated elements

21 Discussion to

determine correct design

20 Continued effort

to complete design

38 Changes to the design

37 Business continues

to change

6 Project late

Figure 4: Partial map showing loop 5

24 Rework of

interrelated elements 21

Discussion to determine correct

design

8 No agreed design

20 Continued effort

to complete design

11 Contractual

negotiations on differences

27 Need to

avoid delay

23 Rework of completed

development

36 33

30

33 38 19 29

32

3 2 6

19

41 40 39 12

16 15 13

10 1

44 Encourage SSS and

AIL staff to work overtime/produce more

34 Disagreement

whether issue is a defect or is

working as designed

35 Many agreed

defects

31 Many (apparent) defects found in

software delivered

Figure 5: Partial map showing loop

24 Rework of

interrelated elements 21 Discussion to

determine correct design

8 No agreed design

20 Continued effort

to complete design

11 Contractual

negotiations on differences

27 Need to

avoid delay 23

Rework of completed

development

44 Encourage SSS and

AIL staff to work overtime/produce more

35 Many agreed

defects

32 Time to perform

testing

6 Project late

1 Supplier late

in delivering design documents

29 Software delivered

with many known defects

30 Software accepted

for testing with many known defects

33

33 31

37 26

25 43 10

19 34 2

16 15 13

41 40 39 12

31

33 38 19

34

Figure 6: Partial map showing loops 1

progress in resolving outstanding

issues (concept 113) led back into dis-

satisfaction (concept 108), and this

increased the desire to reduce negative

impacts (concept 109), it did not then

lead to an increase in the decision to

focus attention on migration (concept

112). Two of the other loops also con-

tain this loop as a subset, and are thus

also spurious.

The only genuine loop found

within this map is loop 14: AIL staff

are highly bonused, so loss of produc-

tivity had a big impact on their satis-

faction. This led to staff developing a

poor opinion of the application and to

demoralization, which affected pro-

ductivity even more. This loop impact-

ed the whole business but particularly

the new business acquisition area

where user confidence helps sales abil-

ity and therefore sales performance,

allowing a destructive positive feed-

back loop to be perpetuated. AIL man-

agement quickly intervened to

minimize the impact of the positive

feedback, primarily through commu-

nication. Project personnel ensured

that operational management within

the areas affected were kept up-to-date

with the likely impacts of application

issues. This allowed management to

develop realistic expectations and

therefore set realistic targets that

helped to counter demoralization

65S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

Central Concepts

Two closely related concepts, concept 7

(introduction of parallel working) and

concept 20 (ongoing effort to com-

plete design), were of central signifi-

cance in the resulting late delivery of

Project X2. They feature in multiple

positive feedback loops, and in fact all

but one of the loops goes through one

or the other (or frequently both) of

these concepts. This is indicative of the

centrality of these issues for the very

significant time extensions experi-

enced during this project. In fact,

chronologically, the failure to com-

plete the design stage led to the intro-

duction of parallel working, and,

therefore, this one concept may rea-

sonably be considered the most signif-

icant point at which an alternative

decision may have affected the entire

course of the project, allowing it to

avoid becoming significantly late. It

should be noted that action at this

point would not have prevented the

time delays completely, as it was

already late by the time this was

observed. However, action could have

prevented the destructive positive feed-

back, observed in the map, from exac-

erbating the existing problems to the

eventual proportions they took.

Outcomes

The outcomes of this map are repre-

sented here as ovals with a shadow and

bold text. These are typically, but not

necessarily, the heads of the map (con-

cepts that have only arrows going in).

This map contains four outcomes: the

increased amount of senior manage-

ment time expended by both parties to

bring the project to completion (con-

cepts 13 and 28), and, most signifi-

cantly, the late delivery of the

application to the business and its

reduced scope when it was delivered

(concepts 6 and 25, respectively).

Obviously, the late delivery meant that

AIL's internal staff costs for project per-

sonnel increased significantly,

although the external cost did not

increase due to the fixed-price contract.

Late delivery also meant that the busi-

ness benefits that comprised the busi-

ness case for the investment could not

be realized until delivery was com-

plete. However, even after completion

the other outcome, reduced scope,

meant that the benefits were not

immediately realizable.

Post-Launch Map

The map drawn to represent the situa-

tion post-launch (Figure 8) is much

simpler than that of pre-launch. It

includes three concepts drawn in from

the previous map (25, 39, and 40): the

scope of the product, and two con-

tributing factors to the scope of the

product. In addition it contains only

23 new concepts.

Decision Explorer identifies four

loops within this map. However, three

of the loops are (temporally) spurious.

One apparent causal loop involved the

decision to focus attention on comple-

tion of the data migration task at the

expense of resolving some of the other

issues. This decision was taken with

the full knowledge that it would have

an impact on the resolution of out-

standing issues, and that this would

likely have an impact upon the user

community. However, it was decided

that this was a necessary compromise

to allow completion of the rollout

phase of the project, because the

impact of operating two applications

simultaneously was greater than the

impacts of the other issues requiring

attention. So that, while the slow

24 Rework of

interrelated elements

21 Discussion to

determine correct design

8 No agreed design

20 Continued effort

to complete design

11 Contractual

negotiations on differences

27 Need to

avoid delay

23 Rework of completed

development

33 Time to analyze

failure

32 Time to perform

testing

41 Introduce

joint-working

6 Project late

1 Supplier late

in delivering design documents

25 Scope of product

36 17

19 38 35 19

34 30

37 26

34 2

44 12

16 15 13

37 26

43

10

40 De-scope: Accept

more defects for go-live

39 De-scope: Remove

data migration

Figure 7: Partial map showing loops 7, 8, 9 & 10

66 S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

developing within the user community. It also allowed them

to set expectations regarding what the user community

would find, and when they could expect issues to be

resolved. This went some way to reducing the negative opin-

ions being formed. These management decisions could not

reverse the effects of the feedback loop, but were able to

minimize them.

There are four tails in this map (likely to be trigger

events) in addition to two of the concepts replicated from

the pre-launch map for context. Like those two, the first also

feeds into the product scope. Following launch, defects that

had not been discovered during the testing phase began to

come to light (concept 101). There were not many of these,

but they further diminished the product's scope. The second

trigger event was the dependence of the rollout of the prod-

uct on data migration (concept 118). At launch, some new

policies were written in the new system; however, even had

100% of new business been written in the new system, there

would remain the preexisting customer base with its records

in the legacy system, so any contact with those customers

would require access to that system. For this reason, and due

to the decision to de-scope data migration from the launch,

rollout could only progress up to a point until data migra-

tion was completed. The third trigger was outside of the con-

trol of AIL: fixing defects and completing the data migration

required the same SSS resources accessing the same code

base (concept 111), so they were constrained by the available

resources from working to resolve the outstanding defects as

quickly as AIL. Finally, the fourth trigger is the primary deci-

sion that helped to minimize the effects of the positive feed-

back established. The introduction of improved

communication channels between the project and the busi-

ness (concept 120), achieved by bringing into AIL an expe-

rienced external consultant for the period leading up to, and

immediately succeeding, the launch of the application.

There are three outcomes new to this map. Two relate to

staff: concept 114 dealing with demoralization and concept

117 dealing with the negative attitude that began to develop

toward the application among people who had not yet

received their training. These were minimized through

improved communication. The third outcome was that the

business suffered at best a failure to realize benefits, and at

worse real losses through the introduction of this applica-

tion. As indicated in the analysis of trigger events, these out-

comes in part derived from the failure to identify all of the

potential problems that existed. However, they also arose

from the decisions made pre-launch to de-scope the appli-

cation through accepting more outstanding defects, com-

promises on functionality and de-scoping the data

migration. These were the decisions that helped to break the

vicious cycles that were preventing the achievement of a

launch date.

109 Desire to reduce negative impacts

and secure benefits

105 Customer service

issues

106 Productivity losses

experienced

39 De-scope: Remove

data migration

118 Completion of rollout

requires data migration

107 Extension to duration

of rollout until all data migration

activity complete

112 Decision to focus

available resource on data migration

113 Slow progress

resolving outstanding issues

111 Limited available

supplier resources 108

End user dissatisfaction with product

103 Effort to train staff

102 Introduction of a

number of workarounds 104

Anticipated productivity gains not realized

25 Scope of product

40 De-scope: Accept more

defects for go-live 101

Previously undiscovered defects found

119 Rollout strategy

requires simultaneous operation of both

legacy and new systems 120

Improved project business community

communications

121 Realistic expectations

among operational management

123 Realistic relaxation

of productivity targets

122 Realistic expectations

set among user community

114 Demoralization within users of new system

115 Poor opinion of application

among workforce

116 Negative word-of-mouth

communication to users of legacy system

110 Adverse impacts on AIL's

profit & loss

117 Staff yet to undergo

training already negatively disposed to change

Figure 8: Post-launch map

“cover their backs,” slowing progress

that might have been achieved in a

more cooperative environment.

Second, AIL decided to make use of the

design documents that resulted from

that first failed project as the initial

baseline for the new project (concept

229); however, few people from AIL

and none from SSS were involved in

that project and no one was complete-

ly comfortable with the documents, so

there was disagreement as to their con-

tent and meaning.

The project manager surfaced an

important outcome not apparent in

the BPM's map. Events in the project

leading up to the launch led to a com-

mercial dispute between the two

companies (concept 251), which

threatened to sour a relationship,

which at times had been fractious, but

which more recently had been improv-

ing with the successful adoption of

joint working. The effects of this

were particularly seen in the post-

launch map.

67S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

Concepts such as 114 dealing with

demoralization within the staff and

concept 117 dealing with the negative

attitude that began to develop toward

the application are “soft” issues, which

are not usually dealt with in project

management bodies of knowledge.

This type of issue is seldom considered

in project reviews, and the feedback

loops that might result may go unrec-

ognized. It is a strength of this

approach that they were captured and

can be linked into more tangible proj-

ect events like the extension of time

for rollout.

Construction of Cognitive Maps for the

Project Manager

Two cognitive maps were also con-

structed following interviews with the

project manager, again showing events

leading up to launch and those follow-

ing. Many of the features found in the

project manager's maps have already

been surfaced in the BPM's maps, as

previously discussed. There are no

contradictions between the two

managers' maps, but there are some

further features in the project

manager's maps that did not

previously appear, some of which are

discussed here.

Analysis of the project manager's

pre-launch map shows a number of

interesting trigger events. One was the

choice of SSS to include a package that

they did not own as part of the solu-

tion; SSS's subsequent decision during

the project to buy the intellectual

property rights for this product was

cited by them as a partial cause of the

delay. A second was AIL's previous

failed attempt at this project with a dif-

ferent supplier. This resulted in some

key AIL management decisions. First,

the legal case launched by AIL against

the supplier, which was settled out of

court, had an effect on both AIL man-

agement and SSS management when

negotiating the contract for this project

(concept 231), so that key people on

both sides felt the need always to

310 Involvement of SSS senior management

309 Involvement of

AIL senior management

311 Introduction of very rigorous

approach

312 Mechanism to

highlight issues

307 AIL request

additional new changes

306 Conflict for

SSS resource

303 Extension of rollout time

313 Keep project

progress in focus

315

301304

323 317

305 Many defects

requiring urgent resolution

316 Lack of agreement

on migration approach

302 Stop joint

working approach

Figure 9: Partial map showing loops 21 and 22

68 S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

The project manager's post-launch map contained just

22 new concepts, plus two reproduced from the pre-launch

map for context. It contained two sets of double loops, all

negative feedback. These loops, shown in Figure 9, are

unique to this map and show the beneficial impact of the

involvement of senior management following launch. As

can be seen from the map, there were several pressures with-

in the project forcing the rollout period to be extended,

while at the same time conflict for the use of limited SSS

resources prevented resolution of all of these. The involve-

ment of AIL and SSS senior management was seen as a key

to keeping the focus, which had been somewhat lost, on

completing the project and shortening the rollout period.

Also, the prioritization of work by the senior management

team helped to resolve some of the conflict for resources.

One further outcome was staff in both SSS and AIL started

to tire of the project, and looked to move on to something

new. This can lead to a drop in morale among the project

team if allowed to go on unchecked.

Recommendations for Projects

The recommendations for AIL can be divided into two: first,

recommendations specific to how projects should be run in

future following analysis of cognitive maps drawn based

upon the experience and perceptions of three key managers

in Project X2; and second recommendations for how AIL

may approach learning from projects in future.

A number of specific recommendations can be made for the

future conduct of Project X2:

1. The business has benefited greatly from effective

communication; care should be taken to ensure that

this momentum is not lost.

2. Steps need to be taken to ensure that the X2 project

team is refreshed, and given new focus for the

remainder of the project to avoid degradation of morale.

For future projects within AIL:

3. Care should be taken to ensure that all design work

has been completed and agreed before work begins

on code development. Failure to do this can result in

much rework, both in areas where the design is sub-

sequently agreed to be other than that developed and

of inter-dependent areas. This does not mean that it

is not possible to complete activities in parallel, just

that care needs to be taken to ensure the independ-

ence of parallel activities.

4. Overtime should be used sparingly and for the reso-

lution of short-term issues-for the supplier as well as

AIL's staff. Extensive use of overtime can lead to

unproductive staff, and poor quality output. The res-

olution of issues caused by poor quality can take

much longer than that saved, especially where the

defect is not discovered quickly and subsequent work

is built upon that defect. 5. Applications should not be accepted from the sup-

plier until successful completion of all (or most) of the agreed testing. Acceptance of the application with a large number of known outstanding defects caused AIL's own testing effort to extend.

6. Every effort should be made to avoid making changes

to an application's design during the project. Some

change may be inevitable; however, the impact to the

rework cycle is significant, the effort being not only

to make the change requested, but also to change all

dependent elements.

7. The joint working initiative should be repeated

where feasible. This was responsible for shortening

the turnaround of defect resolution, freeing up

resources to continue other work, and preventing

excessive rework of interdependent elements.

8. If a project is running late, opportunities to reduce

the scope of the deliverable(s) should be examined;

however, care must be taken to ensure that the

impact upon business is understood and accepted

before doing so.

Learning Recommendations

Any organization that desires to improve its competitiveness

in today's market place must learn from its experiences.

Therefore, in addition to the specific recommendations aris-

ing from Project X2, there are more general recommenda-

tions for the ongoing learning of the organization resulting

from the experience of this learning study:

1. The mechanism of learning adopted should suit the

nature of the project. For simple projects, a straight-

forward lessons-learned process may be appropriate,

but for more complex projects, especially where there

is any suspicion that there may have been positive

feedback effects at work, a mapping approach such as

that adopted here is more effective.

2. Learning should not be left to the end of the project.

It should be integrated into the heart of the project

process, on a regular basis as well as following any

significant event.

3. Cognitive maps can be created through one-on-one

interviews or in a group setting. The decision of

which approach to take should depend on the cir-

cumstances at the time:

• Consider group approaches if time is limited,

and/or if it is essential that all parties can identi-

fy with, and “buy-in” to, the output.

• Consider individual interviews if participants

come from various levels within the organization

and there may be disagreement between senior

and junior participants, and/or some participants

are likely to be significantly more forceful so that

quieter team members may get drowned out.

4. When constructing the cognitive maps, attempt to

identify the management actions taken in reaction to

the events of the project, and all of the consequences

of those actions, intended or otherwise; these can

provide crucial triggers or exacerbating elements of

the explanation of the project outcomes.

5. As well as “hard” quantitative explanations of behav-

ior, look also for the “softer” human-oriented ele-

ments, as these can also provide essential elements of

69S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

the causal chains that provide explanation of the

project behavior.

6. When constructing cognitive maps:

• Look for positive feedback loops, which are like-

ly to be root causes of spiraling costs or time scales.

• Investigate tails and consider if these are truly

root-causes of the effects being or if there might be

something more fundamental that has been missed.

• Investigate any orphans and determine how these

concepts link in to the rest of the material.

• Consider all of the heads. Could these be consid-

ered outcomes? If not, ask if these lead to any-

thing that should be recorded on the map.

Conclusions

Reviewing projects is essential if a business is to learn how

to better conduct projects in future. The main recommenda-

tions arising from this study have been described in the pre-

vious two sections, but this section will summarize the results.

Concerning learning from projects:

• It is critical that managers learn the impacts their deci-

sions might have on a project. Some of these are unex-

pected, so typically are difficult to identify. Cognitive

mapping can help to do this, and can show positive

feedback loops resulting from management decisions,

which can have serious adverse effects on a project.

Managers should learn to recognize these feedback

loops when they are occurring (and similarly for ben-

eficial impacts).

• It is widely agreed that learning should not be left to

the end of a project, but should be incorporated

throughout, at regular intervals, and following signif-

icant events.

• For projects that are not complex, a straightforward

lessons-learned process is likely to be sufficient.

However, this is insufficient when faced with com-

plexity and will not bring out the difficult, and some-

times counterintuitive, lessons. Standard lessons

learned processes are good mechanisms for recording

the lessons already learned so that they may be dis-

seminated further, but they are not themselves good

for creating new insights into the workings of

the project.

As far as Project X2 is concerned, a fundamental prob-

lem was the supplier's original underestimation of the com-

plexity of the task, which early on led to the project

becoming late. In order to draw back some overrun, man-

agement in both AIL and SSS decided to execute parts of the

project in parallel, so that a significant rework cycle devel-

oped in which completed work regularly had to be reexam-

ined, and development work proceeded without agreement

on the design baseline. This resulted in commercial dis-

agreement, confusion, time wasted analyzing false prob-

lems, and a delivery that did not always meet the business's

needs. Later, management tried to increase the project

team's output by the use of overtime; this combined with

the unclear design led to the eventual delivery being of a

poor standard. AIL's management decided to accept the

release despite the known problems. This resulted in a pos-

itive feedback loop, which caused the testing end-date to

move increasingly further away. The testing phase was clear-

ly going to continue to extend unless significant decisions

were taken to resolve the problems. First, part of the delivery

was de-scoped allowing concentration to be focused on the

essentials. Second, joint working was introduced, signifi-

cantly speeding up the resolution of issues and breaking a

feedback loop that had impacted the testing phase. The de-

scoping decision had a downside, as after launch it became

apparent that the business impact of some of the de-scoped

functionality was significant, and affected productivity as

well as the time required to complete the rollout of the

application to all users. This was minimized by a decision to

hire in temporary expertise to ensure sufficient communica-

tion between the business and the project team.

Cognitive mapping is a powerful tool for reviewing

projects:

• It allows identification of chains of causality and feed-

back loops not readily identified by intuition alone.

• It helps direct an interview process to ladder up from

concepts already surfaced to outcomes and to ladder

down to trigger events.

• Care needs to be taken when concepts describe effects

that happened in different time scales; this can lead to

the identification of spurious loops. Identified loops

need to be thoroughly checked for this where the map

describes events over an extended time frame.

Separate maps to represent different phases of the

project might help.

• “Soft” issues such as morale can be incorporated into

the maps, and these issues on the performance of the

project can be examined.

Considering “how does one learn from projects?”

requires a different understanding of the dynamics of proj-

ects, and how those dynamics shape the impacts we as man-

agers have on the course of a project. Even where we do

understand impacts intuitively, these techniques help to give

a rational foundation for instincts and allow the manager to

argue these points more successfully. It was an original

learning objective of this work to learn how to better run a

project; however, it became clear that the most valuable

learning comes from understanding the sometimes counter-

intuitive effects of management decisions and how project

events and decisions are interrelated and interacting. Well-

intentioned management decisions to resolve potential

issues can lead to counterintuitive positive feedback loops,

driving projects further off course rather than

controlling them.

References

Abdel-Hamid, T. K., & Madnick, S. E. (1990). The elu-

sive silver lining: How we fail to learn from software devel-

opment failures. Sloan Management Review 32, 39-48.

70 S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

Baccarini, D. (1996). The concept of project complexi-

ty-A review. International Journal of Project Management 14,

201-204.

Barker, M., & Neailey, K. (1999). From individual learn-

ing to project team learning and innovation: A structured

approach. Journal of Workplace Learning 11, 60-67.

Central Computer and Telecommunications Agency.

(2001). Prince 2: An Outline. London: The Stationary Office.

Cooke-Davies, T. (2002). The “real” success factors on

projects. International Journal of Project Management 20, 185-

190.

Cooke-Davies, T. J., & Arzymanow, A. (2003). The

maturity of project management in different industries: An

investigation into variations between project management

models. International Journal of Project Management 21, 471-478.

Cooper, K. G. (1994). The $2000 hour: How managers

influence project performance through the rework cycle.

Project Management Journal 25, 11-24.

Cooper, K. G., Lyneis, J. M., & Bryant, B. J. (2002).

Learning to learn, from past to future. International Journal of

Project Management 20, 213-219.

Disterer, G. (2002). Management of project knowledge

and experiences. Journal of Knowledge Management 6, 512-520.

Dvir, D., Raz, T., & Shenhar, A. J. (2003). An empirical

analysis of the relationship between project planning and

project success. International Journal of Project Management

21, 89-95.

Easterby-Smith, M., Thorpe, R., & Lowe, A. (2002).

Management Research: An Introduction, Second Edition.

Wiltshire: Sage Publications Ltd.

Eden, C. (1988). Cognitive mapping. European Journal

of Operational Research 36, 1-13.

Eden C, Ackermann, F., & Cropper, S. (1992). The

analysis of cause maps. Journal of Management Studies 29,

309-324.

Eden, C., & Ackerman, F. (1998). Making Strategy.

Journey of Strategic Management. Wiltshire: Sage Publications Ltd.

Eden, C., Williams, T., Ackermann, F., & Howick, S.

(2000). The role of feedback dynamics in disruption and

delay on the nature of disruption and delay (D&D) in major

projects. Journal of the Operational Research Society 51, 291-300.

Garvin, D. A. (1993). Building a learning organisation.

Harvard Business Review 71, 78-91.

Gray, R. J. (2001). Organisational climate and project

success. International Journal of Project Management 19, 103-109.

Howick, S., & Eden, C. (2001). The impact of disruption

and delay when compressing large projects: Going for incen-

tives? Journal of the Operational Research Society 52, 26-34.

Jaafari, A. Project management in the age of complexity

and change. Project Management Journal, 34; 47-57.

Jiang, J. J., Klein, G., Chen, H.-G., & Lin, L. (2002).

Reducing user-related risks during and prior to system devel-

opment. International Journal of Project Management 20, 507-515.

Jørgensen, M., & Sjøberg, D. I. K. (2004). The impact of

customer expectation on software development effort esti-

mates. International Journal of Project Management 22,

317-325.

Journal of Management Development. (2001). Management's

role in project failure. The Journal of Management

Development 20.

Keil, M., & Robey, D. (1999). Turning around troubled

software projects: An exploratory study of the deescalation

of commitment to failing courses of action. Journal of

Management Information Systems 15, 63-87.

Kotnour, T. (2000). Organizational learning practices in

the project management environment. International Journal

of Quality and Reliability Management 17, 393-406.

Lester, S. (1999). An Introduction to Phenomenological

Research. Available online at

http://www.devmts.demon.co.uk/resmethy.htm (8th July

2004).

Lyneis, J. M., Cooper, K. G., & Els, S. A. (2001). Strategic

management of complex projects: A case study using system

dynamics. System Dynamics Review 17, 237-260.

Moynihan, T. (2002). Coping with client-based “peo-

ple-problems”: The theories-of-action of experienced is/soft-

ware project managers. Information & Management 39,

377-390.

Office of Government Commerce. (2002). Managing

Successful Projects with PRINCE2. London: The

Stationary Office.

Project Management Institute. (2000). PMBOK® Guide -

2000 Edition. Newtown Square, PA: Project

Management Institute.

Reichelt, K., & Lyneis, J. (1999). The dynamics of

project performance: Benchmarking the drivers of cost and

schedule overrun. European Management Journal 17,

135-150.

Rodrigues, A., & Bowers, J. (1996). The role of system

dynamics in project management. International Journal of

Project Management 14, 213-220.

Rodrigues, A. G., & Williams, T. M. (1997). System

dynamics in software project management: Towards the

development of a formal integrated framework. European

Journal of Information Systems 6, 51-66.

Rodrigues, A. G., & Williams, T. M. (1998). System

dynamics in project management: Assessing the impacts of

client behaviour on project performance. Journal of the

Operational Research Society 49, 2-15.

Schindler, M., & Eppler, M. J. (2003). Harvesting project

knowledge: A review of project learning methods and suc-

cess factors. International Journal of Project Management 21,

219-228.

Shenhar, A. J., Dvir, D., Levy, O., & Maltz, A. C. (2001).

Project success: A multidimensional strategic concept. Long

Range Planning 34, 699-725.

Smith, H. J., Keil, M., & Depledge, G. (2001). Keeping

mum as the project goes under: Towards an explanatory

model. Journal of Management Information Systems 18, 189-227.

Turner, J. R., & Cochrane, R. A. (1993). Goals-and-

methods matrix: Coping with projects with ill defined goals

and/or methods of achieving them. International Journal of

Project Management 11, 93-102.

White, D., & Fortune, J. (2002). Current practice in project

71S E P T E M B E R 2 0 0 6 P R O J E C T M A N A G E M E N T J O U R N A L

STEPHEN ROBERTSON, PhD, was educated at the Universities of Glasgow and Strathclyde. He worked as a post-doctoral

research associate at the University of Liverpool, conducting research into the thermodynamics of DNA super-coiling. A

career change took him into a large insurance company, where he worked in various operational management roles before

taking on a role managing major projects in the organization. He has had that role for seven years.

TERRY WILLIAMS, PhD, PMP, was educated at the Universities of Oxford and Birmingham, U.K., and has worked in

operations research (OR) for nine years at BAESYSTEMS, developing project risk management work, and later acting as

risk manager. He is currently at a professor at Southampton University. Previously he was professor of OR and head of

department at Strathclyde University. He is a speaker and writer on modeling in project management, and has written

more than 60 articles in refereed OR and project management journals and a book titled "Modelling Complex Projects."

He is a member of numerous research networks worldwide, being particularly involved with PMI, where he is one of the

Research Members Advisory Group. He edits the Journal of the Operational Research Society and sits on a number of

other journal advisory boards.

management-An empirical study. International Journal of

Project Management 20, 1-11.

Williams, T. (1999a). The need for new paradigms for

complex projects. International Journal of Project Management

17, 269-273.

Williams, T. (1999b). Seeking optimum project dura-

tion extensions. Journal of the Operational Research Society 50,

460-467.

Williams, T. (2000). Safety regulation changes during

projects: The use of system dynamics to quantify the effects

of change. International Journal of Project Management 18, 23-31.

Williams, T. (2002). Modelling Complex Projects.

Chichester: John Wiley & Sons Ltd.

Williams, T. (2003a). Learning from projects. Journal of

the Operational Research Society 54, 443-451.

Williams, T. (2003b). Assessing extension of time delays

on major projects. International Journal of Project Management

21, 19-26.

Williams, T. (2004). Identifying the hard lessons from

projects-Easily. International Journal of Project Management

22, 273-279.

Chichester: John Wiley & Sons Ltd.

Williams, T. (2003a). Learning from projects. Journal of

the Operational Research Society 54, 443-451.

Williams, T. (2003b). Assessing extension of time delays

on major projects. International Journal of Project

Management 21, 19-26.

Williams, T. (2004). Identifying the hard lessons from

projects-Easily. International Journal of Project Management

22, 273-279.