RA-4

Assessing Scope and Managing Risk in the Highway Project Development Process

Tiendung Le, S.M.ASCE1; Carlos H. Caldas, A.M.ASCE2; G. Edward Gibson Jr., P.E., F.ASCE3; and Michael Thole4

Abstract: The project development process is a critical component of highway projects. Decisions made during this phase have a significant impact on the final project outcomes. This paper describes a research project that studied this process and subsequently developed a method to help the highway project team improve the project development process. This method does so by proactively identifying risk sources based on the analysis of the project scope. This method uses a comprehensive list of scope elements with descriptions and a mechanism to evaluate quantitatively the scope elements’ level of definition. Assessing the level of definition of each scope element and of the project as a whole allows the project team to determine the potential level of risk to which the project is exposed. The project team can then develop risk mitigation plans to respond to the potential high risk elements. The proposed method was tested on real completed and ongoing projects undertaken by experienced professionals. The method was well received by the subject matter experts, and a number of benefits were observed, including the use as an integrated checklist, a mechanism for monitoring the project development progress, as well as a means for improving communication and promoting alignment within the project team.

DOI: 10.1061/�ASCE�CO.1943-7862.0000052

CE Database subject headings: Construction management; Planning, Highway and road construction; Risk management.

Introduction

The project development process �PDP� is strategically important for highway projects. It aims to assure that the right project is selected and adequately planned for the subsequent project phases. The PDP requires careful and detailed coordination among all elements involved in the project who are involved in such tasks as planning and programming; design; environmental assessment; right-of-way acquisition; utility adjustments; plans, specifications, and estimates development; construction; and maintenance.

Because project scope definition and risk management are critical components of the PDP, a method that can help facilitate and improve the definition of project scope elements is desirable. Improving project definition is a proactive approach to risk man-

agement because it allows for addressing risks at their sources. Such a method needs to include the broad range of issues across disciplines while emphasizing the interactions among them. It should provide sufficient details about the scope elements while maintaining the big picture of the entire PDP.

The advance planning risk analysis �APRA� tool and method were developed to help the owner’s project team overseeing highway projects to manage proactively risk during the PDP by directly identifying risk sources. The method includes a compre- hensive list of elements that the project team needs to address during the PDP. �The term element is used interchangeably with APRA element, risk element, and scope element in this paper.� The early planning elements, if not proactively and properly ad- dressed early in the project, are potential sources of risk.

This paper reports on the entire process of developing and testing the APRA. It starts with a review of the literature related to the method and an argument for the need for it. It then provides details on the development phase followed by a section on how to use the method. The paper continues with the presentation of the testing of the APRA on real completed and ongoing projects. Initial benefits of the method were captured during the testing and are presented in the section that follows. Finally, the paper dis- cusses some limitations and avenues of future research.

Background

A typical highway project’s life cycle includes six main phases as illustrated in Fig. 1. The PDP is the period that covers all of the four first phases of the project life cycle, from needs assessment to detailed design. Closely related to the PDP is “advance plan- ning,” which refers to the process that includes the first three phases �needs assessment, feasibility/scoping, and preliminary de- sign�. Advance planning has several acronyms; the most fre-

1Ph.D. Candidate, Dept. of Civil, Architectural, and Environmental Engineering, Univ. of Texas at Austin, 1 University Station C1752, Austin, TX 78712. E-mail: [email protected]

2Assistant Professor, Dept. of Civil, Architectural, and Environmental Engineering, Univ. of Texas at Austin, 1 University Station C1752, Austin, TX 78712 �corresponding author�. E-mail: caldas@mail. utexas.edu

3Professor, Garry Neil Drummond Endowed Chair, Dept. of Civil, Construction, and Environmental Engineering, Univ. of Alabama, 259 H.M. Comer �MIB� Box 870205, Tuscaloosa, AL 35487-0783. E-mail: [email protected]

4Project Controls Engineer, Oil, Gas, and Chemicals Unit, Bechtel Corporation, 3000 South Post Oak Blvd., Houston, TX 77056. E-mail: [email protected]

Note. This manuscript was submitted on May 23, 2008; approved on February 2, 2009; published online on April 30, 2009. Discussion period open until February 1, 2010; separate discussions must be submitted for individual papers. This paper is part of the Journal of Construction Engineering and Management, Vol. 135, No. 9, September 1, 2009. ©ASCE, ISSN 0733-9364/2009/9-900–910/$25.00.

900 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009

quently used ones are preproject planning, front-end planning, and conceptual planning. Such planning is defined by the Con- struction Industry Institute �CII 1994� as “the process of develop- ing sufficient strategic information with which owners can address risk and decide to commit resources to maximize the chance for a successful project.” It is an important subset of project planning and it is typically the responsibility of the owner �Gibson et al. 1995�. The early intensive involvement of major project stakeholders with diverse expertise �e.g., planning, design, environmental, right-of-way, and construction� is required if the project’s objectives are to be effectively met. The advance plan- ning and PDP in relation with the entire project life cycle are illustrated in Fig. 1.

The PDP is a “long-lasting, comprehensive, and complex pro- cess” �Arts and Lamoen 2005�. During these early project stages, the scope is defined and refined. Efforts invested during this pe- riod have far more influence on project success than those during the construction phase �Gibson et al. 1995�. Therefore, strategies and techniques that can streamline the PDP have the potential to improve the project performance.

The need to improve the PDP has been emphasized by recent studies that indicated poor cost and time performance of transpor- tation infrastructure projects. A research investigation looking at 258 infrastructure projects �i.e., roadways, rail, fixed links� world- wide reported that 90% of the projects experienced cost overrun with an average cost escalation of 27.6%. The escalation for road- way projects was 20.4% �Flyvbjerg et al. 2003�. One may argue that the performance of a later phase �i.e., construction� is not solely dependent on the performance of the earlier phases, how- ever, to a large extent it does depend on such earlier performance �CII 2008a,b�. In spite of this connection, much more attention in project management research and practice has been paid to con- struction, while much less has been focused on the PDP �Atkin- son et al. 2006; Arts and Lamoen 2005�.

As one of the areas of project management and a key practice in the advance planning process, risk management is no exception to this lack of attention. The current practices and literature tend to focus on the management of risk events. This approach does not consider the range of risk sources in a project �Atkinson et al. 2006�. An intensive literature review of project risk management by Williams �1995� has also shown that most of the research on risk management focus on risk events. Similarly, project risk management processes developed and used by various organiza- tions tend to address risk as an event, such as those of the Cali- fornia Department of Transportation �Caltrans�, the Federal Highway Administration �FHWA�, and the Association of Project Managers in the U.K. �California Department of Transportation 2003; Ashley et al. 2006; Chapman 1997�.

An approach that identifies and addresses risk at its source is a more proactive one that a project team can use. And as more than one risk can originate from one source, addressing a risk source may give the project team the power to solve the root of the

problems effectively, while managing more than one risk at a time. For instance, poor soil conditions may seriously affect the construction schedule and also impact the efficacy of the founda- tion or base of a roadway. Understanding the implications of the existing soil conditions in advance planning will help address potential risks in this area before detailed design and construction through design or contracting language. Moreover, it is not un- common for risks that are previously unknown by the project team to arise. Addressing risk sources may help the project team, proactively or even unknowingly, prevent the occurrence of such unknown risks.

Systematically assessing project scope elements is probably one of the most appropriate ways to identify risk sources because scope elements are those that altogether describe the entire work that the project team needs to perform. By completely defining all the work included in the scope and described by these elements, the project team essentially addresses risk sources, and thereby minimizes risks that may originate from the lack of satisfactory management of these sources. This approach has been success- fully used in a number of efforts in the past, such as those by CII �2008a,b�.

There have been various studies focused on project scope defi- nition and management during the PDP. One of the first methods developed and used extensively in the heavy industrial construc- tion sector was that by Hackney �1965�. To rate the state of project scope definition quantitatively, Hackney proposed a method based on checklists. In all of Hackney’s checklists, each item is assigned a weight, which depends on the item’s relative importance to the project.

The project definition rating index �PDRI� is a successful ad- vance planning method developed by CII �2008a,b�, Cho and Gibson �2001�, and Dumont et al. �1997� for assessing project scope definition during the front-end planning of building and industrial projects. Front-end planning in building and industrial construction is similar to advance planning in highway construc- tion. The PDRI has a list of project scope elements, including descriptions, which are organized in categories and sections. Using a rating mechanism for each element’s definition, the PDRI allows the project team to determine the level at which a pro- ject is defined at any given time during the front end planning process.

In the highway construction sector, Shane �2006� developed a scope definition index for use in the early project development of design-build highway projects. Shane’s results include a check- list of attributes and a mechanism for weighting them on a scale from one to six. This research focuses on design-build projects, which are normally let with less than 30% design �Shane 2006�. This level of design is usually obtained at the completion of preliminary design. This research, while having an important contribution for scope definition in the early phase of project development, does not address scope definition from the end of preliminary design to the completion of detailed design, an

Fig. 1. Project development and advance planning processes in project life cycle

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009 / 901

equally important period that involves high level of activity and complexity in all major functions such as environmental process, right-of-way acquisition, utility relocation, and design.

The review of the current state-of-the-art reveals the impor- tance and need for methods that can help improve the overall effectiveness of the PDP in highway projects using scope defini- tion. Industrial and building construction sectors have successful methods for scope definition in front end planning. There is also research on scope definition in early phase of design-build high- way projects. The current literature, however, lacks a method and tool for scope definition of the entire PDP, from needs assessment to detailed design completion. The APRA was developed to meet this need. The research methodology was adopted from similar successful research efforts by �CII 2008a,b� and from the work made by Hackney �1965,1992�.

The results from structured interviews with eleven Texas De- partment of Transportation experts at the beginning of the re- search reaffirmed the need for a method like the APRA. One of the main objectives of these interviews was to understand the current processes and practices, including the existence and use of tools that the practitioners had their disposal during the PDP. Al- though most of the disciplines had developed tools to help expe- dite the planning process and keep track of progress, these tools did not cover all the requirements in the disciplinary work. The interviewed experts repeatedly requested a new method that broadly covered relevant issues �Thole 2006�. The feedback from the interviewed experts was, therefore, a practical motivation for a method like the APRA.

Development of the Advance Planning Risk Analysis

The APRA method uses an integrated list of scope definition el- ements that altogether describe the work that needs to be per- formed during the project development process. Each of the scope elements has a description of its parameters and what specific tasks or issues it may include. The method uses six definition levels to represent how much is known about each element at a given point in time. The overall project scope definition is as- sessed by considering the definition of all individual elements. By knowing which scope elements are not well defined, the project team can proactively identify potential risk sources. The team thus can develop action plans to respond to and work to mitigate these potential risks. This section will present how the project scope elements were identified, categorized, and weighted based on their potential impact on a project.

Identifying and Categorizing Risk Elements

The first step in developing the APRA was to identify the scope elements that the project team needs to address during the entire project development process. The methodology used to perform this identification was to investigate current published state DOT and federal processes and literature related to the highway project development processes, as well as to interview experts with ex- tensive relevant experience. A variety of sources of information and practices were reviewed to generate a preliminary list of scope elements. These sources were also used for developing the descriptions of each element and the items �i.e., specific tasks and issues� pertaining to each element. The documents and sources used included manuals for various functions performed during the project development processes of state departments of

transportation, such as those of Texas and Minnesota DOT �Texas Department of Transportation �TxDOT� 2000, 2003, 2004a,b, 2005a,b,c,d; Minnesota Department of Transportation �MnDOT� 2002�; publications by federal agencies and institutions such as the FHwA �2000, 2001, 2002�, Transportation Research Board �Waters 2000� and the CII �2008a,b�; and input from periodic meetings with the DOT project monitoring committee, which was in charge of monitoring, guiding, and assisting the research team.

In generating a list of elements, the research team also con- ducted face-to-face interviews with eleven professionals who had relevant experience and expertise in this subject matter. This in- vestigation allowed the researchers to identify critical project de- velopment elements that are of practitioners’ concern.

As a result, a list of 59 elements was generated in addition to their respective descriptions, which provide essential information about each of the elements, their significance to the project, and the considerations they require. The complete list and descriptions of all these elements are included in Caldas et al. �2007a�. An example of one of the 59 elements’ descriptions, “B4. Future expansion and alteration considerations,” is displayed in Fig. 2. The elements cover the major tasks that need to be performed in all main areas of the highway project development process. The tasks relate to different phases of the project life cycle and involve all project stakeholders, including federal and state agencies.

The next step was to synthesize and categorize the list of ele- ments into meaningful and organized groups. After a series of internal research team meetings, the 59 elements were segregated into twelve categories, which were further grouped into the fol- lowing three sections: basis of project decision, basis of design, and execution approach. The sections represent their relative se- quences in project development while the categories are groups of related tasks. The elements, categories, and sections are presented in the two left columns in Appendices I–III.

Advance Planning Risk Analysis Element Definition Levels

The description of a project scope element provides the level of detail and the tasks that need to be performed for that element. Early in the project development process, the project scope ele- ments are not well defined. However, as the project moves along

Fig. 2. APRA element’s description example

902 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009

to the later phases and more tasks have been performed, the project scope elements become more defined. The definition level is therefore used to indicate the level at which each element is defined at a given point in time in comparison with its complete definition. A scale of five levels, from one to five, is used for this purpose. Additionally, a definition level of “zero” is used to indi- cate an element that is not applicable to the project being as- sessed. The definition levels are described as follows: • Level 1: completely defined. The element is well defined. All

of the work pertaining to the element is performed completely. No more work is required.

• Level 2: minor deficiencies. Only some minor work is needed for several items of the element.

• Level 3: some deficiencies. There is major work needed for some items or some work needed for most of the items of the element.

• Level 4: major deficiencies. There is major work needed for most of the items described in the element.

• Level 5: incomplete or poor definition. The element is poorly defined. Major work is needed for all or almost all items of the element. As the descriptions reveal, Definition Level 1 is the most de-

sired status of an element while Definition Level 5 is the least preferred. This preference is not meant to imply that level five is bad since it also depends on the time of the assessment to judge an element’s definition level.

Weighting the Advance Planning Risk Analysis Elements

Although all scope elements are critical to the development of a project, they have different relative impacts on the project. An element with a higher impact would pose a higher risk to the project’s success if it is not properly addressed. Therefore, more attention should be paid to those with higher relative impacts.

The relative impacts of scope elements are not obvious. These impacts should reflect the practices of the project development process, therefore expertise in project development should be used to weigh the elements. Among the weighting methods con- sidered, the research team determined that using workshops with experienced professionals would be the most suitable way to evaluate the elements’ relative importance, to have direct interac- tions with the participants, to maximize information conveyance consistency, and to improve the response rate. An alternative method would have been to capture actual definition level data for all 59 elements and compare to project performance, thereby de- veloping factor weightings. Because of the large data pool that this method would require and the length of development time required, among other considerations, the writers chose not to use this method for development.

Six workshops were organized with the participation of 51 experts. The participants were from all major disciplines in the project development process, including programming and plan- ning, design, right-of-way acquisition, utility adjustment, environ- mental, and surveying. The participants’ experiences ranged from a few years to more than 30 years. At the workshops, the experts were asked to select a typical project among those they had been involved in and use it as a reference project in the entire weight- ing process that would follow.

For each element there were two scenarios. First, if the ele- ment, as described in the scope element descriptions document provided, was incompletely or poorly defined �Definition Level 5�, respondents were asked how much contingency they would

assign to that element. Both time and cost effects as the result of poor definition of the element were taken into consideration; both types of effects should be converted to monetary value, in terms of a percentage of the project’s total installed cost. The second scenario was the case when the element was completely defined �Definition Level 1�. It is logical that when the element is more defined, less contingency should be assigned to it to offset the uncertainties it may bring to the project during its execution. This process was used for all elements on the list. The participants were given some time at the end of the workshop to adjust the contingency values.

The contingencies assigned for the poorly defined case of an element would be used to calculate the score for Definition Level 5 of that element. This score was the maximum score an element could have and it denoted the weight of the element versus other elements in the tool. The more weight an element had, the more important it was to a project. Likewise, the contingencies for the well defined case were for calculating the score of Definition Level 1. Note again that Level 1 was the desired level of defini- tion when an element was well defined. However, the score of Level 5 determined the importance of an element.

It was not unusual for an element to be not applicable in a project regardless of size. In this case, the expert was asked to write “N/A” in both places for levels of Definitions 1 and 5 of that element.

Calculating Weights of the Elements’ Definition Levels

The next research step was to calculate the elements’ weights based on the data collected from the workshops. Out of these 51 weighting forms received from the 51 workshop participants, two had a significant amount of missing data, and thus were dis- carded. Three other participants had less than 3 years of experi- ence so their completed forms were considered unsuitable for use in calculating the elements’ weights. After this preliminary data screening, data sets from 46 experts were qualified to be included in the further data analysis. Their experience had a wide range of distribution, from 3–31 years with an average of 17.7 years. Five of the participants have less than 10 years of experience, 25 with 10–20 years, and 16 with more than 20 years.

A score range from 70–1,000 points was selected to represent the final score of a project. The project score is obtained by add- ing up the scores of all elements. A score close to or at 1,000 indicates a very poorly and incompletely defined, and therefore, highly risky project. On the other hand, a score close to 70 means that the project is well defined. The score of 70 as a minimum was chosen because it corresponded to weighting scales used in the PDRI tools developed by CII. A project has a maximum score of 1,000 when all elements are applicable and have Definition Level 5.

The contingencies assigned for elements �in percentage of total installed cost� by the workshop participants were used to determine the elements’ score at Definition Levels 1 and 5. The percentage values at Level 5 were normalized so that the total of all the elements’ scores at Level 5 for each participant is 1,000 points. The boxplot technique was used to screen the normalized scores. Participants with a significant number of element scores that were outliers were discarded from further data analysis. As a result, data sets from 39 participants were valid for use in calcu- lating elements’ scores. An element’s scores at Level 5 as as- signed by all participants were averaged to become the score of that element. These scores were rounded and adjusted to be inte- gers and added up to 1,000 points.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009 / 903

After being normalized, scores at Level 1 were adjusted so that their total was 70 points. The value of 70 signifies the fact that there is still risk in a project even if all elements are deemed to be well defined, i.e., having Definition Level 1.

From the element scores at Definition Levels 1 and 5, scores at Levels 2, 3, and 4 were determined using linear interpolation. As mentioned before, Level 0 was used to denote the case when an element was not applicable to the project. A nonapplicable ele- ment would have Level 0 and be eliminated from any consider- ation and further analysis of the project. A project with nonapplicable elements would have a total maximum possible score of less than 1,000 points. Weights of all elements were finalized and are presented in Appendices I–III.

Analyzing Weighting Results

An element has the highest score when it has a definition level of 5. This highest score represents the importance of the element; the higher the score, the more important the element is to a project. A category has the maximum score when all of its elements have their maximum scores. This maximum score also illustrates the relative importance of the category when compared with other categories. Likewise, the highest scores of all categories in a sec- tion will make the section have the maximum score. And maxi- mum section scores add up to the project maximum score, which is 1,000. Fig. 3 shows the weights of all categories and sections.

Interestingly, the weights of the three sections are similar, from 30% total weight for Section I to less than 36% total weight for Section II. This implies that in a highway project, the basis of project decision, basis of design, and execution approach contrib- ute relatively equally to the outcome of the project. Section

I—basis of project decision, consists of information necessary for understanding the project objectives. The completeness of this section determines the degree to which the project team will be able to achieve unification in meeting the project’s business ob- jectives. Section II—basis of design, consists of geotechnical, hy- drological, environmental, structural, and other technical design elements that should be evaluated to understand fully the design’s impact. Finally, Section III—execution approach, consists of ele- ments that should be evaluated to understand fully the require- ments of the owner’s execution strategy and approaches for detailed design, right-of-way acquisition, utility adjustments, and construction.

The ten most highly weighted scope elements are presented in Table 1. These elements’ total weight accounts for 25% of that of all elements �250 out of 1000�. These elements are the most criti- cal ones to the project development process from the workshop participant’s perspective. If poorly defined, they will have the biggest negative impact on the outcome of a project. Conversely, if well defined, these elements will play a large role in securing the success of the project. It is not necessarily implied that con- centrating on the ten most critical elements will ensure project success; rather, the suggestion is that while more attention should be paid to the most important ones, all of the elements need to be properly addressed.

How to Use the Advance Planning Risk Analysis

The APRA should be used at various points during the project development process especially prior to moving to a subsequent stage or at any time of critical importance. As mentioned previ- ously, the project development process covers all stages from the initiation of the project to the beginning of construction. Fig. 4 illustrates the points in time when the APRA should be applied.

Table 1. Ten Most Highly Weighted Scope Definition Elements

Rank Element

ID Weight Element

1 C4 30 Determination of utility impacts

2 A3 30 Programming and funding data

3 C3 26 Survey of existing environmental conditions

4 A2 25 Investment studies and alternative assessments

5 I1 24 Long-lead parcel and utility adjustment identification

6 E3 24 Schematic layout

7 B1 23 Design philosophy

8 A1 23 Need and purpose documentation

9 A5 23 Public involvement

10 D5 22 Environmental documentation

TOTAL 250

Fig. 3. APRA section and category weights �at Definition Level 5�

Fig. 4. Employing the APRA, application points

904 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009

Evaluating a project using the APRA should be performed in a team environment, such as a meeting, if its benefits are to be best used. Each meeting, especially the first one for a project, should be facilitated by a neutral person who is knowledgeable about team dynamics and familiar with the method. A meeting may take from 2–4 h depending on the familiarity of the participants with the APRA method.

A project’s key participants �including those from right-of- way, utilities, design, planning, environmental, and construction� should participate in the evaluation meeting not only to provide input for the evaluation, but also to gain insight into the issues of concern to others. This multidisciplinary participation is useful for the project team as it helps enhance collaboration among the members. At the meeting, each APRA element is evaluated and assigned a consensus level of definition based on the current knowledge the team possesses and the work done pertaining to that element. After determining the elements’ definition level, the team will be able to identify what should have been done, what will be done, what might go wrong, and who should take care of each issue. All of the information from discussion should be cap- tured in a report that provides the basis for risk mitigation.

When evaluating a project, the team assesses each element based on how much work has been done pertaining to the element as compared to the element’s description at the time of the assess- ment. A definition level is selected to reflect this assessment. Each level of each element has a predetermined score �as presented in Appendices I–III�. Once all elements have been assessed and as- signed a definition level, their scores are tallied to get scores of corresponding categories, sections, and the entire project. Ele- ments that are not applicable to a specific project can be zeroed in upon, thus allowing their elimination from the final scoring cal- culation. However, during the evaluation, scores should not be the major focus. Focusing too much on the scores may undermine the APRA’s main benefit, which is helping the project team as a whole identify the risky areas of the project and develop a plan of action to improve the project.

At the end of this evaluation, the team may continue with discussion of mitigation actions, or plan another meeting to de- velop an action plan in response to the high risk elements using the list generated and the notes captured during the evaluation. All of the information should be developed into a risk register for the project team to control, manage, and communicate the risk elements. Using the APRA at different times during the project development process help the project team monitor the progress it makes between the evaluations. Details on how to use the APRA method and its companion computer tool can be found in the corresponding project research reports �Caldas et al. 2007a,b,c�.

The APRA, while essentially being a scope definition method, can be used for managing risk. However, it should be used in conjunction with other traditional risk management methods and techniques, which can help determine specifically attributes of risk such as likelihood of occurrence and level of impact. The APRA elements �and the relevant work items in their descrip- tions� and the evaluation mechanism can be used for identifying risks. The information recorded during evaluation meetings and the elements’ weights can serve as a good basis for the project team to determine the likelihood and potential impacts of risks. Also, based on the information and facts from the evaluation an action plan can be developed in response to the risks identified. And last, the continuous evaluation of the project at critical phases would allow the project team to monitor risk over time.

A significant benefit of the APRA is that it allows using the

APRA score to predict the likelihood of project success based on experience. This benefit can be realized once the APRA has been used for a significant number of projects. Based on the APRA scores and project performance data, one can perform analysis on the relationship between project score and project performance. This relationship can be used to predict project performance of future projects if the project score is known. Therefore, the project score would then be an indicator of the general level of risk that a project is involved.

Testing of the Advance Planning Risk Analysis

The APRA was tested on real projects to verify its viability as a method and tool. The testing allowed the research team both to get feedback from practitioners and to understand how the APRA works in a real project environment as well as what benefits it brings to a project. Both completed and ongoing types of projects were selected for testing the APRA. A completed project is a project which has construction that is completely finished and the assessment will be retroactive. An ongoing project is a project that has not yet been let and may be at any point prior to letting.

For each project, the testing process involved participants from various functional areas who were involved in planning the project. Prior to testing each project, the participants were asked to fill in the background and performance information for the project. Each testing session was performed in a meeting that lasted for about 2 1/2 h and was facilitated by one or two of the writers. The participants were asked to use definition levels to rate how well each element was defined at the beginning of the de- tailed design phase if the project was complete and at the current time if it was ongoing. A total of 32 project participants were involved in testing 16 sample projects.

The APRA was tested on 16 projects. Of these 16 projects, one sample project did not have sufficient basic background informa- tion and was therefore discarded. The sample of 15 projects, eight completed and seven ongoing, included five types of highway projects: interchange, new location freeway, new location non- freeway, widened freeway, and widened nonfreeway. The projects’ final total installed costs ranged from approximately $3.8 million to nearly $104.7 million with an average of about $29.1 million. This group of projects, though limited in number, represented a wide range of projects types and sizes.

At the end of each assessment, the APRA tool automatically generated a list of low definition elements for each of the projects. These are the high risk elements. Different projects had different characteristics, were developed by different project teams, and thus had different lists of high risk elements. However, there are elements that appeared more often in all high risk lists, such as survey of existing environmental conditions �C3�, determination of utility impacts �C4�, hydrological characteristics �D2�, hydrau- lic structures �F2�, and long-lead parcel and utility adjustment identification �I1�.

Using “survey of existing environmental conditions” as an ex- ample, in one of the tested projects, the project engineers who had been involved in the project told the authors that this part of the work had been thought to have been well defined during project development. Later on, during construction, it turned out that there was a gas line under the to-be-built bridge. The contractor drilled into the active gas line and caused a serious leak. An area of 1 sq mi �2.6 sq km� around the area had to be evacuated and isolated. Fortunately, the leaked gas was not ignited and there were no fatalities. Had the APRA been used to assess the project

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009 / 905

during the project development process, the likelihood of this occurrence could have been minimized since the use of subsur- face utility engineering is listed in the description of surveys and planimetrics �D3� as an important aspect to be considered for the completeness of the scope definition of this APRA element.

The two most important objectives of testing the APRA on real projects were to observe how the APRA would work in a real project environment and to obtain feedback from experts who participated in the testing process. The experts provided numer- ous comments on the APRA and its potential use. These com- ments were used in drawing initial conclusions on the potential APRA benefits that are presented in the section “Benefits of the Advance Planning Risk Analysis.” In addition to the request to provide comments throughout the test, at the end of each test meeting the participants were also asked to give opinions on two specific propositions: �1� the APRA method helps identify critical elements that need to be managed during the project development process and �2� the APRA method helps improve the project de- velopment process. A Likert scale was used for both propositions. The experts could choose any level from 1 �strongly disagree� to 7 �strongly agree�.

All participants agreed that the APRA method helped identify critical elements that need to be managed during the project de- velopment process. Its checklist format is the most obvious ad- vantage of the APRA as it is helpful and easy to use. The list of high risk elements that is identified at the end of each assessment also provided practical information to the project team. Most of the experts �27 out of 32� agreed that the APRA could help im- prove the project development process; four of them were neutral on the proposition; and one disagreed. This result shows great potential for the APRA to be brought into practice. Further dis- cussions with the experts and the analysis of their comments re- vealed the experts’ insightful understanding of the method and the project development process. Some have commented that the tool itself was good but implementing it would raise certain barriers, such as the users’ reluctance to adopt the method as a result of perceived added workload. Also, it was observed that top man- agement needs to support the implementation of the APRA since one of the biggest challenges in the project development process and in the utilization of methods such as the APRA is getting people from different disciplines collaboratively involved.

With the small sample of projects tested, there was not suffi- cient data to draw reliable and meaningful conclusions on the relationship between the level of scope definition, represented by the APRA score, and the project’s performance. The testing pro- cess was, therefore, largely qualitative. The use of a project’s APRA score to interpret the level of project scope definition and to predict the project’s performance would be enabled when there are sufficient number of projects that have been evaluated using the APRA.

Benefits of the Advance Planning Risk Analysis

A project development team can use the APRA to identify, ana- lyze, control, and manage risks across disciplines �e.g., planning and programming, design, environmental engineering, right-of- way acquisition, and utilities adjustment�. A significant feature of the APRA is that it can be used to fit the needs of highway projects of different sizes. The APRA is both quick and easy-to- use. It is a method that can provide numerous benefits to different participating organizations of the project development process.

The APRA elements along with their corresponding descrip-

tions provide an integrated checklist of elements whose definition is critical to the project development process. Its format as a checklist provides a significant advantage to a project develop- ment team since it combines all major issues of concern with a common set of definitions in one document. Such a format is the APRA’s most obvious benefit and it has been well recognized and received by the APRA test participants.

Using the APRA to rate the completeness of each scope ele- ment helps a project team identify the sources from which risks can arise as well as develop action plans. The project team can also monitor the project development progress by using the APRA at different times in project development and comparing assessment results. The team then can develop proper action plans based on the progress.

One of the significant concerns among the project develop- ment stakeholders is how to communicate across disciplines and how to reconcile differences. Using the APRA to evaluate project development in a team setting allows the project team members to communicate the issues within their functions to people of other disciplines and probably discuss strategies to tackle the issues. Open communication can help promote team alignment since team members know more of others’ concerns and objectives.

An important potential benefit is the APRA’s use as a bench- marking tool for organizations. This use of the APRA will be enabled after it has been used for some time when sufficient num- ber of projects have been evaluated and the evaluation and project performance data have been recorded for analysis. The past project performance and APRA evaluation data can be analyzed to predict future project performance based on APRA evaluation results, including the APRA score, of the project.

Research Limitations and Extensibility

In the development of the APRA, a wide range of literature sources have been reviewed to generate the list and descriptions of the risk elements. The list is, therefore, generic and can be used widely for different types of organizations in different regions. When determining the weights of the elements, only experts working in Texas were involved. Even though the project devel- opment process is similar in most states, the elements’ weights may not best reflect the circumstances and conditions of other regions. Thus, when used in the regions where project develop- ment practices are significantly different from those of Texas, the elements should perhaps be reweighted, or validated in some form. If reweighted, it is recommended that workshops should be used as a method for reweighting the elements. The APRA method and its usage would remain unchanged.

Conclusions

This paper presented the development and testing of the APRA method, which is an innovative tool that can help a project team to improve the highway development process through scope defi- nition and proactive management of risk. Fifty nine elements de- scribe the scope of work that needs to be defined during the project development process. These elements were weighed based on their relative importance to a project using input from experi- enced professionals. A scale of six levels was used to evaluate each element’s definition in terms of scope of work.

The APRA, in its most basic form, can be used as an inte- grated checklist of critical scope elements that need to be defined

906 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009

by the project development team. The scoring mechanism allows the project team to assess the level of scope definition a project has before moving to the letting and construction phases. The use of this scoring process at different points in time along the project development process provides the project team with a method for benchmarking their own efforts as the project evolves. And be- cause the APRA takes into consideration work in all disciplines involved, it creates a unique platform for the project team mem- bers from different disciplines to communicate, cooperate, and collaborate effectively. Team alignment can therefore be im- proved. The APRA tool and list of elements can be extended and used by different departments of transportation and municipalities thanks to the consideration of a broad range of scope elements identified in relevant publications and documents during the re- search study.

The APRA method has been tested on a limited number of completed and ongoing projects with promising results and strong positive feedback. Further testing is needed. It is anticipated that after being used in a sufficient number of projects, data from the APRA evaluation and project performance can be analyzed to verify potential correlations between level of scope definition and project performance.

Acknowledgments

The writers thank the Texas Dept. of Transportation for its finan- cial support. They would also acknowledge the support of the project monitoring committee. Michael Thole is a former gradu- ate student at The University of Texas at Austin.

Appendix I. Advance Planning Risk Analysis Weighted Project Score Sheet—Section I

Section I—Basis of Project Decision

Category element

Definition level

Score0 1 2 3 4 5

A. Project strategy �maximum = 122� A1. Need and purpose documentation 0 1 7 12 18 23

A2. Investment studies and alternatives assessments 0 2 8 14 19 25

A3. Programming and funding data 0 2 9 16 23 30

A4. Key team member coordination 0 1 6 11 16 21

A5. Public involvement 0 2 7 13 18 23

Category A total

B. Owner/operator philosophies �maximum = 76� B1. Design philosophy 0 1 7 12 18 23

B2. Operating philosophy 0 1 5 10 14 18

B3. Maintenance philosophy 0 1 5 9 12 16

B4. Future expansion and alteration considerations 0 2 6 11 15 19

Category B total

C. Project requirements �maximum = 102� C1. Functional classification and use 0 1 5 8 12 15

C2. Evaluation of compliance requirements 0 1 6 10 15 19

C3. Survey of existing environmental conditions 0 2 8 14 20 26

C4. Determination of utility impacts 0 2 9 16 23 30

C5. Value engineering 0 1 4 7 9 12

Category C total

Section I maximum score = 300 Section I total

Note: Definition levels: 0 = not applicable; 2 = minor deficiencies; 4 = major deficiencies; 1 = complete definition; 3 = some deficiencies; and 5 = incomplete or poor definition.

Appendix II. Advance Planning Risk Analysis Weighted Project Score Sheet—Section II

Section II—Basis of Design

Category element

Definition level

Score0 1 2 3 4 5

D. Site information �maximum = 173� D1. Geotechnical characteristics 0 1 5 9 12 16

D2. Hydrological characteristics 0 1 5 10 14 18

D3. Surveys and planimetrics 0 1 5 10 14 18

D4. Permitting requirements 0 1 5 9 13 17

D5. Environmental documentation 0 2 7 12 17 22

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009 / 907

Section II—Basis of Design

Category element

Definition level

Score0 1 2 3 4 5

D6. Property descriptions 0 1 5 8 12 15

D7. Ownership determinations 0 1 4 7 10 13

D8. Right-of-way mapping 0 1 5 9 12 16

D9. Constraints mapping 0 1 6 10 15 19

D10. Right-of-way site issues 0 1 6 10 15 19

Category D total

E. Location and geometry �maximum = 79� E1. Horizontal and vertical alignment 0 1 6 11 15 20

E2. Control of access 0 1 5 9 13 17

E3. Schematic layouts 0 2 8 13 19 24

E4. Cross-sectional elements 0 1 5 10 14 18

Category E total

F. Structures �maximum = 48� F1. Bridge structure elements 0 1 5 9 12 16

F2. Hydraulic structures 0 1 5 10 14 18

F3. Miscellaneous design elements 0 1 4 8 11 14

Category F total

G. Design parameters �maximum = 29� G1. Provisional maintenance requirements 0 1 4 6 9 11

G2. Constructability 0 1 5 10 14 18

Category G total

H. Installed equipment �maximum = 30� H1. Equipment list 0 1 3 5 7 9

H2. Equipment location drawings 0 1 3 5 6 8

H3. Equipment utility requirements 0 1 4 7 10 13

Category H total

Section II maximum score = 359 Section II total

Appendix III. Advance Planning Risk Analysis Weighted Project Score Sheet—Section III

Section III—Execution Approach

Category element

Definition level

Score0 1 2 3 4 5

I. Acquisition strategy �maximum = 137� I1. Long-lead parcel and utility adjustment identification 0 2 8 13 19 24

I2. Long-lead/critical equipment and materials identification 0 1 4 7 9 12

I3. Local public agencies utilities contracts and agreements 0 1 6 10 15 19

I4. Utility agreement and joint-use contracts 0 1 6 11 15 20

I5. Project delivery method and contracting strategies 0 1 4 7 10 13

I6. Design/construction plan and approach 0 1 4 8 11 14

I7. Procurement procedures and plans 0 1 3 6 8 10

I8. Appraisal requirements 0 1 4 8 11 14

I9. Advance acquisition requirements 0 1 4 6 9 11

Category I total

J. Deliverables �maximum = 23� J1. CADD/model requirements 0 1 3 6 8 10

J2. Documentation/deliverables 0 1 4 7 10 13

Category J total

K. Project control �maximum = 98� K1. Right-of-way and utilities cost estimates 0 2 7 12 16 21

K2. Design and construction cost estimates 0 2 7 12 16 21

K3. Project cost control 0 1 5 9 13 17

K4. Project schedule control 0 1 5 9 12 16

908 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009

Section III—Execution Approach

Category element

Definition level

Score0 1 2 3 4 5

K5. Project quality assurance and control 0 1 3 6 8 10

K6. Safety procedures 0 1 4 7 10 13

Category K total

L. Project execution plan �maximum = 83� L1. Environmental commitments and mitigation 0 1 5 8 12 15

L2. Interagency coordination 0 1 5 8 12 15

L3. Local public agency contractual agreements 0 1 5 8 12 15

L4. Interagency joint-use agreements 0 1 4 8 11 14

L5. Preliminary traffic control plan 0 1 4 7 10 13

L6. Substantial completion requirements 0 1 4 6 9 11

Category L total

Section III maximum score = 341 Section III total

References

Arts, J., and Lamoen, F. V. �2005�. “Before EIA: Defining the scope of infrastructure projects in The Netherlands.” J. Environ. Assess. Policy Manage., 7�1�, 51–80.

Ashley, D. B., Diekmann, J. E., and Molenaar, K. R. �2006�. “Guide to risk assessment and allocation for highway construction manage- ment.” Rep. No. FHWA-PL-06-032, Federal Highway Administration, Office of International Programs, Washington, D.C.

Atkinson, R., Crawford, L., and Ward, S. �2006�. “Fundamental uncer- tainties in projects and the scope of project management.” Int. J. Proj. Manage., 24�8�, 687–698.

Caldas, C. H., Gibson, G. E., and Le, T. �2007b�. “TxDOT best practices model and implementation guide for advance planning risk analysis for transportation projects.” Texas Dept. of Transportation Research Rep. No. 0-5478-P2, Center for Transportation Research, Univ. of Texas at Austin, Austin, Tex.

Caldas, C. H., Gibson, G. E., and Le, T. �2007c�. “User guide for the advance planning risk analysis tool for transportation projects.” Texas Dept. of Transportation Research Rep. No. 0-5478-P1, Center for Transportation Research, Univ. of Texas at Austin, Austin, Tex.

Caldas, C. H., Le, T., Gibson, G. E., and Thole, M. �2007a�. “Identifying right-of-way requirements during the project development process.” Texas Dept. of Transportation Research Rep. No. 0-5478-1, Center for Transportation Research, Univ. of Texas at Austin, Austin, Tex.

California Department of Transportation. �2003�. “Project risk manage- ment handbook.” Rep. Prepared for the California Dept. of Transpor- tation, Office of Project Management Process Improvement, Sacramento, Calif.

Chapman, C. �1997�. “Project risk analysis and management—PRAM the generic process.” Int. J. Proj. Manage., 15�5�, 273–281.

Cho, C.-S., and Gibson, G. E. �2001�. “Building project scope definition using project definition rating index �PDRI�.” J. Archit. Eng., 7�4�, 115–125.

Construction Industry Institute �CII�. �1994�. “Pre-project planning: Begin a project the right way.” Research Summary Rep. No. 39-1, Construction Industry Institute, Univ. of Texas at Austin, Austin, Tex.

Construction Industry Institute �CII�. �2008a�. “Project definition rating index, building projects, 3rd Ed.” Implementation Resource Rep. No. 155-2, Construction Industry Institute, Univ. of Texas at Austin, Austin, Tex.

Construction Industry Institute �CII�. �2008b�. “Project definition rating index, industrial projects, 3rd Ed.” Implementation Resource Rep. No. 113-2, Construction Industry Institute, Univ. of Texas at Austin, Austin, Tex.

Dumont, P. R., Gibson, G. E., and Fish, J. R. �1997�. “Scope management using the project definition rating index �PDRI�.” J. Manage. Eng., 13�5�, 54–60.

Federal Highway Administration �FHwA�. �2000�. “Project development guide.” Federal Highway Administration, Washington, D.C., �http:// www.fhwa.dot.gov/realestate/pdg.htm� �Oct. 24, 2008�.

Federal Highway Administration �FHwA�. �2001�. “Real estate acquisi- tion guide for local public agencies.” Federal Highway Administra- tion, Washington, D.C., �http://www.fhwa.dot.gov/REALESTATE/ lpaguide/index.htm� �Oct. 24, 2008�.

Federal Highway Administration �FHwA�. �2002�. “European right-of- way and utilities best practices.” Rep. Prepared for Federal Highway Administration, FHWA-PL-02–013, Office of International Programs, FHwA, Washington, D.C.

Flyvbjerg, B., Holm, M. K. S., and Buhl, S. L. �2003�. “How common and how large are cost overruns in transport infrastructure projects?” Transplant. Rev., 23�1�, 71–88.

Gibson, G. E., Jr., Kaczmarowski, J. H., and Lore, H. E., Jr. �1995�. “Preproject-planning process for capital facilities.” J. Constr. Eng. Manage., 121�3�, 312–318.

Hackney, J. W. �1965�. Control and management of capital projects, Wiley, New York.

Hackney, J. W. �1992�. Control and management of capital projects, 2nd Ed., K. K. Humphreys, ed., McGraw-Hill, New York.

Minnesota Department of Transportation �MnDOT�. �2002�. “Project de- livery streamlining: Design, right-of-way, environmental focus areas.” Rep. Prepared for Program Delivery Streamlining Task Force, MnDOT, Minneapolis.

Shane, J. S. �2006�. “A fundamental study of scope definition in early highway project development.” Ph.D. dissertation, Univ. of Colorado at Boulder, Boulder, Colo.

Texas Department of Transportation �TxDOT�. �2000�. Right-of-way considerations training course (DES 110) student manual, TxDOT, Austin, Tex.

Texas Department of Transportation �TxDOT�. �2003�. Project develop- ment process manual, Design Division, TxDOT, Austin, Tex.

Texas Department of Transportation �TxDOT�. �2004a�. Environmental manual, Environmental Affairs Division, TxDOT, Austin, Tex.

Texas Department of Transportation �TxDOT�. �2004b�. Hydraulic design manual, Design Division, TxDOT, Austin, Tex.

Texas Department of Transportation �TxDOT�. �2005a�. Right of way manual, Right of Way Division, TxDOT, Austin, Tex.

Texas Department of Transportation �TxDOT�. �2005b�. Roadway design manual, Design Division, TxDOT, Austin, Tex.

JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009 / 909

Texas Department of Transportation �TxDOT�. �2005c�. PS&E prepara- tion manual, Design Division, TxDOT, Austin, Tex.

Texas Department of Transportation �TxDOT�. �2005d�. Utility manual, Right of Way Division, TxDOT, Austin, Tex.

Thole, M. �2006�. “Advance planning in highway project development processes: Key variable identification and categorization.” MS thesis, Univ. of Texas at Austin, Austin, Tex.

Waters, T. �2000�. “Innovative practices to reduce delivery time for right- of-way in project development.” Rep. Prepared for National Coop- erative Highway Research Program, Transportation Research Board, Washington, D.C.

Williams, T. �1995�. “A classified bibliography of recent research relating to project risk management.” Eur. J. Oper. Res., 85�1�, 18–38.

910 / JOURNAL OF CONSTRUCTION ENGINEERING AND MANAGEMENT © ASCE / SEPTEMBER 2009