Project Mgnt Essay

US_Census_FDCA_Case_Study_V1.0.pdf

Home >Psychology homework help >Project Mgnt Essay

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

Failed Project Case Study – The United States 2010 Decennial Census – Field Data Collection Automation (FDCA) A Project Management case study by Calleam Consulting Ltd

Synopsis Efforts to develop more efficient ways to conduct the USA’s decennial national census get derailed when a project to introduce handheld computing devices turns sour. Despite the promise of improved efficiency and cost savings, requirements and technical performance issues identified through dress rehearsals bring deep seated management problems to a head. The resulting problems force the Census Bureau to abandon core components of the program. The resulting redesign and associated increase in the need for manual labour to conduct the census adds $3B in costs to the execution of the 2010 census.

Background It’s an administrative task on the grandest of scales. Taking ten years to plan, a full year to execute and months of analysis to complete, the decennial census of the United States of America is a mind boggling undertaking. Requiring the participation of every adult living in the United States and drawing on the efforts of 1.3 million temporary workers [1], the 2010 decennial census was the United States largest peacetime mobilization. Under Article 1, Section 2 of the US constitution a census of all persons living in the United States must be conducted once every ten years. The data collected becomes a critical element in the planning of both government and private activities. Among other things, census data is used to allocate seats in the US House of Representatives and for the annual allocation of $182B in federal funds to states [2]. Given the money involved and the data’s role in establishing the political map of America, accuracy of the data collected is of paramount concern. The task of producing accurate data is the mandate of the US Census Bureau. Located in the Washington DC area the department oversees the census’s multi-‐billion dollar budget and manages a suite of programs that contribute to the preparation and execution of the census. Planning for the 2010 census stated as early as year 2000. One of the first steps in the process was an examination of the outcomes of the 2000 census and the problems that had been encountered along the way. The resulting lessons left planners with two immediate problems.

Y e a r 2 0 0 0 U S N a t i o n a l C e n s u s a t a g l a n c e 1. More than 300 million people

counted 2. 1.5 billion pieces of paper handled 3. 398 million forms Printed and

handled 4. 20 million maps processed 5. 5.8 million phone enquiries from the

public answered 6. 5 million square feet of space in

more than 500 locations across the USA rented

7. 860,000 temporary positions created & staffed by more than 500,00 workers

8. 16,000 phone lines , 15,000 PCs, 1,300 servers and 165 high speed optical scanners installed

9. 42 million in person interviews conducted with those who failed to submit mail-in census forms.

Source: Presentation to the Industry Advisory Council Executive – J. Waite, Mar 17, 2005

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

The first was a matter of cost. Even allowing for inflation, the cost of the census was rising disproportionately with the growth in population. In year 2000 adjusted dollars, the $6.5B cost of the year 2000 census was double that of the 1990 census and for the 2010 census costs were forecast to rise to between $10B and $12B [3]. The rising costs were drawing the attention of the US Congress and the Bureau was asked to find ways of reducing costs. The second issues concerned process. The increasing population and its growing diversity were stretching the limits of the census as it had traditionally been designed. Both the 1990 and 2000 census had encountered difficulties and the accuracy of the data produced was increasingly being called into question. Despite improvements made for the year 2000 census, a 2001 report by PricewaterhouseCoopers estimated that 6 million people had been missed and 3.6 million had been double counted [4].

The Field Data Collection Automation Project The task of addressing the issues fell on the shoulders of Associate Director and Chief Operating Officer for the 2010 decennial census, Preston “Jay” Waite. Waite was a career officer in the Census Bureau and in 2001 he was assigned full and direct responsibility for overseeing the preparations for the 2010 census. Waite had extensive experience within the Census Bureau and in all regards had the experience and training to be considered an expert in census taking [5]. Soon after taking office, Waite recognized that if the Bureau was to meet its goals, the existing paper based census needed to be redesigned. He recognised that increasing the level of automation used during the census could simultaneously improve accuracy and reduce costs. Despite his own admitted lack of technical knowledge [6] Waite decided to equip the bureau’s 525,000 field workers with GPS enabled, on-‐line, handheld computing devices instead of the traditional clipboard and paper. Significant portions of the census workload could then be automated, processes could be streamlined and the data recorded could be more readily integrated into the necessary databases. The handheld devices were to be used for two primary functions,

1. The pre-‐census address canvassing activity in which census workers check the location of each home in the USA so that the Census Bureau’s master address list can be verified

2. The post census Non-‐Response Follow-‐Up (NRFU) interviews that needed to be conducted with individuals who fail to respond to the mail-‐in survey forms.

In principle the concept made sense, GPS would improve accuracy in the agency’s address database and efficiencies could be gained when locating the millions of people needed to be contacted during NRFU. In large part those efficiencies were to come from the fact that field workers could be fed real-‐time updates when a non-‐responder mailed in a late response. This would make interviews more efficient and cut down on the wasted effort spent contacting people who missed the original cut-‐off date, but had submitted a late response. Given the number of people sending in forms late, the benefits were judged to be considerable. To implement his ideas, Waite initiated a program designed to leverage the new technology. The program represented one of the most ambitious programs of change in the Bureau’s history. Between the years 2001 and 2004 Bureau staff worked on establishing the overall business, logical and physical

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

architectures for the redesigned census [7, 8, 9]. One of the most critical elements in the program was a project titled the Field Data Collection Automation project (FDCA). The FDCA project was focused on the deployment of handheld devices and the development of systems to support the address canvassing and NRFU functions. Included in the FDCA project scope were;

1. Acquisition of the handheld devices 2. Development of software to support: address canvassing, workflow management for field

workers (such as assignment of people needing to be contacted as part of NRFU and tracking of progress) and on-‐line forms to allow data collected from non-‐responders to be uploaded

3. Operations infrastructure for regional and local census offices including services such as logistics, training, and help desk support for 12 regional centers, more than 450 local census offices, and up to 500 000 field staff [11]

Costs were estimated to be $800 million and the associated systems, processes and procedures were to be ready for initial field testing in 2004 and full operational testing in the 2006 dress rehearsal.

Chronology of the FDCA project

Table 1 – FDCA timeline

2010 US Census Field Data Collection Automation Timeline

2001 Jay Waite is appointed Associate Director for the Decennial Census. 2001 Waite decides that handheld computing devices will be used for address canvassing and

NRFU activities. Process of redesigning census processes to leverage technology is initiated.

2002 -‐ 2003 Based on commercially available systems, an in-‐house prototype is created by the Census Bureau’s Technologies Management Office.

Feb 2004 Prototype is tested during initial field testing. Significant problems are identified including: Inability to handle the large amounts of data being processed, system freezes, data transmission problems, memory overload and difficulties with mapping features. Following a review the Census Bureau determines that it lacks the necessary technical capabilities or management resources to create a fully functional system [11].

4 Mar 2004 MITRE corporation (an independent not-‐for-‐profit technical consultant) is engaged to prepare an independent assessment on the feasibility of using handheld devices.

28 Jul 2004 Based on research and observations of the prototype in field testing, MITRE completes an “Independent Assessment of the Use of Handheld Computers for the 2010 Decennial Census”. Report indicated handheld computing is feasible, but significant risks exist [10].

Jul – Oct 2004

MITRE assists the Census Bureau in creating a FDCA Statement of Work for use in tendering. MITRE also develops plans and procedures for an FDCA Project Management Office (PMO) [10].

Jan 2005 In part due to slow progress, the FDCA project is reorganized. Responsibility for FDCA is transferred from the Field Directorate to the Decennial Census Directorate. A new Project Manager is assigned and a Project Management Office (PMO) established [11].

Jan 2005 MITRE’s support to the FDCA program is cancelled after the new Project Manager is

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

appointed [10]. Feb 2005 MITRE is re-‐engaged but only to prepare an independent cost estimates for the FDCA

project [10]. Feb 2005 The Census Bureau begins initial information exchange with potential suppliers [11]. Apr – May 2005

Pre-‐solicitation notice and draft solicitation paperwork are published [11].

Jun 2005 The final Request For Proposal (RFP) is published [11]. May 2005 – Jan 2006

Logical and physical architectures for the FDCA system are developed along with initial project management plans (risk management, communications plan, etc) [7, 8, 9].

Sep 2005 – Jan 2006

Qualified, interested parties work with the Census Bureau to [11] to develop prototypes of the proposed system as part of the bidder evaluation process.

30 Mar 2006 $595M contract is signed with Harris Corp of Melbourne, Florida to supply 525,000 devices and develop the software. Work begins on detailed requirements analysis.

Mar 2007 Harris reports “previously established milestones & budgets were at risk due to the unanticipated number of actual requirements emerging from detailed requirements decomposition”. In response Census Bureau re-‐engages MITRE and requests they perform a “Red Team” analysis of the FDCA program [10].

Apr 2007 Address canvassing dress rehearsal shows that the devices suffer from performance, data transmission and stability problems (problems not dissimilar to those encountered during the 2004 testing). In addition, the Bureau determines that the system does not fully meet its functional needs.

Apr 2007 MITRE is asked to assess the functionality of the devices and their performance. 6 Jun 2007 “Red Team” report is published. Report establishes that “FDCA is at significant risk of

cost and schedule overruns and omission of essential requirements unless major changes are made quickly”. Report firmly establishes that problems are the fault of the government and not Harris [10].

29 Nov 2007 MITRE meets with Waite and his superior (Census Bureau Deputy Director) to review status. Meeting notes are leaked revealing that there is no integrated plan, communications have broken down, parties are unsure of the requirements and that on the Bureau’s side the project “lacks a leader with the experience, stature or passion to make FDCA successful” [13].

16 Jan 2008 The Bureau refocuses themselves and Harris is provided with a list of 400 new or changed requirements for the handheld devices.

6 Feb 2008 Census Bureau Director steps in and forms an FDCA Risk Reduction taskforce. Taskforce identifies mitigation strategies and ultimately recommends handheld devices be dropped from NRFU and that work be started on the design of systems needed to support a paper based NRFU.

5 Mar 2008 Following a review the Government Accountability Office designates the FDCA project as being at “High Risk” of failure.

3 Apr 2008 The US House of Representatives is informed of the failure and the Bureau’s recommendation that paper based non-‐response follow-‐up be conducted. An additional $3B in funding is requested.

15 Apr 2008 Waite is replaced by Arnold Jackson who instigates regular meetings between stakeholders and arranges for Harris and Census staff to collocate into a single office space to improve communications on the remaining parts of the project (address canvassing).

23 Apr 2008 Waite retires from the Census Bureau.

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

Causes of the failure On the surface automation of a census appears to be a relatively simple task. Despite the high number of people involved and the geographical distribution of the work, a census is largely an exercise in the collection of relatively simple sets of data. Waite himself used to say in public presentations that although the census involved large numbers of people, the actual census processes themselves are relatively simple [2]. Even following the debacle, a Senior Vice President at MITRE, echoed the sentiment saying "I've been part of complex technology programs and this is not one of them...this is not hard to do" [14]. Of course complexity is relative to the level of prior experience an organization has with such projects and to the Census Bureau the change from a paper based system to a technology enabled one represented the type of large project that they had not previously experienced. Through the years 2001 and 2008, the Census Bureau began to understand that the changes they were making were significantly more challenging that they had first perceived them to be. Following the collapse of the program in April 2008, when asked by House Appropriations subcommittee chairman U.S. Rep. Alan Mollohan (D-‐ W.Va.) the source of the problems, Waite responded “It was clear to me that there wasn’t sufficient scepticism” [15]. Waite’s succinct response provides the key to understanding the causes of failure. Despite the simplicity of the processes in question, moving from paper to handheld computing still involved major challenges. While those challenges were all technically achievable, they still needed to be adequately understood and managed through to conclusion if the project were to be a success. Based on the information available in the public domain, and Waite’s own comments, it appears that the management of the Bureau fell into the trap of assuming that because the business processes themselves were simple, the project itself should be relatively simple too. That attitude set the context within which management approached the project and resulted in a series of mistakes that ultimately snowballed in a catastrophe. At the time the failure was made public in 2008, press reports focused on two issues as being the root causes of the failure; a failure to establish effective communications between stakeholders in the Census Bureau and the project team at Harris and the Bureau’s failure to stabilize a firm requirements baseline. Such mistakes were clearly avoidable. Given that the system was simply the automation of an existing paper based process, the requirements were indeed knowable had sufficient emphasis been placed on proper requirements elicitation. In addition, had proper governance, control and tracking processes been put in place effective communications could have been established. Information available from the leaked MITRE meeting notes and the US Government Accountability Office (GAO) indicate that the problems went beyond those two basic causes and included [6, 13, 17]:

1. Failure in the early years of the project to establish an appropriate governance structure that would bring control to the project and provide management with visibility so that issues could be surfaced and addressed early,

2. Failure to put in place a Project Manager with the appropriate experience of large-‐scale systems development projects,

3. The organization’s inability to establish clear lines of authority for decision-‐making, 4. Ineffective business analysis (the failure to establish the connections between operational needs

and system requirements),

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

5. Failure to establish system performance requirements (and associated acceptance criteria) or to develop tests to ensure those benchmarks could be achieved by the systems being developed,

6. The Project Manager’s failure to build an effective project schedule and to integrate the dates for work in the Census Bureau with those for work being done by Harris,

7. Failure to perform a technical or operational risk analysis early enough in the project, and, 8. The failure to take the lessons learned from the 2004 field trials and to manage those as risks

during the full scale development efforts undertaken by Harris.

A failure to heed the warnings As well as the general Project Management issues outlined above, the FDCA case study also raises another important question; why did the Bureau fail to heed the many warnings that were issued along the way? Dress rehearsals and tests physically demonstrated that there were very real problems. As for many large federal governmental projects, the performance and status of the project was also monitored by the GAO. Through their published reports and testimony to congress the GAO continually warned of project’s serious problems. The themes that emerged from those early warnings closely matched the eventual causes that lead to the failure. Among the themes raised through those reports were the fact that [16];

1. The Bureau needed to define specific measurable performance requirements for the handheld mobile computing device

2. The Bureau needed to develop an integrated and comprehensive plan to control costs and manage the project

3. The Bureau needed to maintain diligent oversight of its contractors; and 4. The Bureau needed to strengthen its systems testing and risk management activities.

Many of these warnings given to the Bureau were quite specific and spelled out precisely where the problem areas were. Among the earliest warnings were the following comments published following an evaluation of the 2004 field testing [17]:

“Transmission problems and inadequate help desk support were the main reasons for the serious disruption of the NRFU operation and will require the design of alternative approaches for future tests and the 2010 Census. … Census needs to plan contingencies for essential NRFU components, like transmissions, whose failure would jeopardize field operations. … Although contracting can help bring the necessary system and software development expertise and management discipline, Census still faces tremendous challenges in capturing lessons learned from the 2004 and subsequent tests; defining complete and verifiable system requirements; preparing the solicitation; selecting a competent contractor; and overseeing the contract so that systems are fully developed, tested, and finalized before operations begin.”

Similar warnings and lengthy reports detailing the issues were published in 2005, 2006 and 2007. Although documents from GAO note that Census Bureau staff had acknowledged the issues, clearly the issues were never adequately addressed. Due to the failure of the Bureau to take the necessary corrective actions the GAO escalated the issues in 2008 by designating the project as “high-‐risk”. While management’s underestimation of the complexity of the project when it first began may be partially

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

understandable (in light of the fact that the business processes being automated were relatively simple), failing to heed the warnings that were published along the way is harder to comprehend. The publically available documents do not tell us why the Bureau failed to heed the warnings. As a result, some conjecture is required. From my own experience I’ve seen a number of projects in which warnings were given, but the senior management responsible failed to take action. Sometimes that was because they lacked confidence in the group raising the concerns. Other times it has been because there are so many fires burning that they didn’t have the time to fully comprehend the seriousness of the issues being raised. Oftentimes I’ve seen management rationalize their way out of the situation. To do they that reject any criticisms raised and convince themselves that critics are naysayers or simply trying to protect their own reputations rather than giving credible advise. For example, managers sometimes take the attitude that auditors are raising concerns simply to justify their existence and that any concerns they raise are overstated “just in-‐case”. As it stands the real reasons the Bureau failed to take appropriate corrective actions remains unknown. Was it because they were too busy fire-‐fighting to sit back and see the big-‐picture unfolding around them? Was it a lack of faith in the GAO’s capabilities that lead the Bureau to dismiss their advise? Or did the Bureau simply not know how to implement the recommendations? We don’t currently know. Maybe with time more information will be made available and we can filling in the missing gap.

Conclusions and recommendations The FDCA project illustrates what is perhaps the most classic of the classic mistakes in Project Management: The under-‐estimation of complexity (click link for additional examples). Although the business process being implemented by FDCA were relatively simple, the introduction of technology introduced layers of risk and complexity that had not been present in the paper based system (data transmission, security, systems performance, etc). In a distributed computing environment such as FDCA (especially one that needed to be used across the full geography of the USA) such challenges can be very considerable. Based on the actions the Census Bureau took it appears that the full scale of that challenge was only understood after considerable effort had already been expended and a good portion of the available time used up. Those delays directly resulted in the project being placed under schedule pressure and that pressure may well have resulted in the failure to establish the project requirements sufficiently before initiating the tendering process. Beyond that fundamental mistake the case study also raises questions as to why Waite failed to establish an effective governance and leadership structure until it was clear the project was already getting into trouble. A Project Management Office (PMO) was put in place, but it was not done until Jan 2005. By that time the Bureau had already tried to complete the project themselves and the requirements and RFP were already underway. In his role as Chief Operating Officer for the census he would have been too senior to actively manage the project himself. His responsibility was to establish a valid management structure within which the project could be properly controlled. Had an effective structure been in place from the project’s inception in 2001 it is likely that many of the problems could have been avoided and those issues that did arise would have had someone to take care of them rather than being ignored for so many years. Again the Census Bureau is not alone in making this mistake. The failure to establish appropriate governance structures (click for examples) is another common theme seen in many project failures. In

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

part the causes of such failures often comes down to a lack of training. As managers rise up the ranks they tend to take responsibility for larger and larger projects. Unfortunately few organizations provide training to their managers in how to deal with these projects and each manager is left to invent their own set of governance process. Managers who have a background in delivering large projects often have a good sense for what to do. Managers who come from the operational side of the organization often lack Project Management experience and as a result, fail to see what steps they need to take in order to establish control over their projects. Waite had a background as a statistician (more of a technical subject matter role) and in operations, but a review of available information about his background does raise questions as to whether he had sufficient Project Management experience to know how to create an effective governance structure for a project of this type (i.e. one with a significant IT component). If he did, the conclusion we must draw is that he did not use that knowledge in this particular project for some reason. What seems more likely is that he lacked the required knowledge and hence didn’t see the need to address such management issues until the weaknesses in the project caused the project to collapse in 2008. The case study suggests that all Senior Managers should be provided training in how to establish governance over a project and what it takes to make a project a success. Senior executives are often the ones sponsoring projects and in that role they have ultimate responsibility for ensuring they have set the project on the road to success. To do that effectively Senior Managers need to: know the basic ingredients needed for a successful project, they need to know what skills and knowledge to look for in the Project Managers they appoint and they need to understand their part in making a project a success. In addition, they need to have enough knowledge to be able to adequately oversee their Project Managers and to be able to tell if a Project Manager is doing an effective job or is out of their depth. Sadly few organizations provide such training and as a result many organizations unwittingly leave themselves open to similar risks. As a final recommendation the case study suggests that Senior Managers need to heed the warnings from the “singing canaries”. In most project failures there are warnings and signs of distress that are apparent long before the final coup-‐de-‐grace occurs. Staff may be expressing concerns, milestones may be getting missed, auditors may be sounding the alarm or there may just be a sense that the project is not under proper control. In their role as project sponsor Senior Managers overseeing projects need to carefully tune themselves into these warnings and recognise when the warnings are genuine signs of problems rather than just background chatter that the Project Manager is able to deal with themselves. Failing to see the signs or choosing to ignore them (as happened in the FDCA project) represents a lost opportunity to take corrective actions sufficiently early. At the end of the day the 2010 US Census did of course go ahead and the herculean administrative task was completed successfully. Waite’s vision was only partially realized (the system was used for address canvassing, but not for NRFU) so the mighty pen, paper and clipboard continued to play its role. The underlying business need for accuracy and efficiency still exists and it is likely that the team planning the 2020 census will need to revisit the NRFU piece once again. Hopefully this time round they can learn the lessons and land a successful project in time for the dress rehearsals to begin again. Robert Goatham Principal Calleam Consulting Ltd Sessional Instructor University of British Columbia Director Wideman Education Foundation Aug 2012

US 2010 Census – Why Projects Fail Case Study -‐ Calleam Consulting Ltd

Calleam Consulting Ltd Calleam Consulting is a training and consulting organization specializing in the fields of Project Management and Project Leadership. A provider of training content and educational services, Calleam works with leading Universities, academic institutions and private organizations to delivery high quality training programs that meet the needs of today's fast paced businesses. Specializing in online learning, Calleam uses cautionary tales from past failed projects to help organizations prepare for the challenges they will face tomorrow. Visit us at Calleam Consulting Ltd. Footnote

1. As I’m writing this case study I’ve been watching the unfolding story of the G4S security shambles at the London 2012 Olympic Games. While not a technology project, the G4S does still have some parallels to the Census Bureau. Again it’s a relatively simple project (hire, train and deploy security guards), but once more, the scale of the project and its distributed nature have proven to be more complex than G4S seems to have realized.

References [1] For 2010 Census, counting gets tougher – H. El Nasser – USA Today, 8 Oct 2008 [2] 2010 Census Planning and Development – P. Jay Waite, Associate Director for Decennial Census – Industry

Advisory Council Executive Luncheon, Washington, DC -‐ 17 Mar 2005 [3] 2000 Census Lessons Learned for Planning a More Cost-‐Effective 2010 Census – Government Accountability

Office, Oct 2002 [4] PricewaterhouseCoopers projects $4.1 billion federal funding loss in 31 states, $3.6 billion for 58 counties – US

Census Monitoring Board press release, 7 Aug 2001 [5] Jay Waite Bio, Census Bureau website, 19 Apr 2007 [6] Features on the Brink – A. Holmes – Government Executive, 15 Jul 2007 [7] 2010 Census Architecture, Volume II: Physical Architecture, Version 1.0 – Jan 31, 2006 [8] 2010 Census Business Architecture, Version 1.0 – Census Bureau, 3 Oct 2003 [9] 2010 Census Architecture, Version 1.1 – Census Bureau, 24 Aug 2004 [10] 2010 Census: Progress on the Development of the Field Data Collection Automation (FDCA) Program and the

Decennial Response Integration System (DRIS) – MITRE, 9 Apr 2008 [11] Final inspection report OSE-‐17368, US Department of Commerce, Inspector General Report, Aug 2005 [12] House Committee on Oversight and Government Reform and the Subcommittee on Information Policy,

Census, and National Archives , April 9, 2008, 2010 Census: Progress on the Development of the Field Data Collection Automation (FDCA) Program and the Decennial Response Integration System (DRIS), J. Providakes, Ph.D. Senior Vice President and General Manager Center for Enterprise Management The MITRE Corporation

[13] Talking points for Meeting with Jay Waite, November 29, 2007 -‐ MITRE [14] Mismanagement, not technology, caused Census handheld trouble, auditors say – G. Nagesh , April 9, 2008 [15] U.S. Census post-‐mortem: An absence of scepticism – Patrick Thibodeau, Computerworld.com blog April 04,

2008 [16] Chronology of Warnings about the Census Bureau’s Field Data Collection Automation System – Rep. H.

Waxman, Chairman, Committee on Oversight and Government Reform, undated [17] Department of Commerce, Office of Inspector General, U.S. Census Bureau — Improving our Measure of

America: What the 2004 Census Test Can Teach Us in Planning for the 2010 Decennial Census (Sept. 2004) (OIG-‐16949).