report 1

THE ENGINEERING ECONOMIST , VOL. , NO. , – http://dx.doi.org/./X..

Using mean-Gini and stochastic dominance to choose project portfolios with parameter uncertainty

Guilherme Augusto Barucke Marcondesa,b, Rafael Coradi Lemeb, Marcela da Silveira Lemeb, and Carlos Eduardo Sanches da Silvab

aNational Institute of Telecommunications, Santa Rita do Sapucaí, Brazil; bInstitute of Industrial Engineering and Management, Federal University of Itajubá, Itajubá, Brazil

ABSTRACT Although a variety of models have been studied for project portfo- lio selection, many organizations still struggle to choose a potentially diverse range of projects while ensuring the most beneficial results. The use of the mean-Gini framework and stochastic dominance to select portfolios of research and development (R&D) projects has been gaining attention in the literature despite the fact that such approaches do not consider uncertainty regarding the projects’parameters. This article dis- cusses, with relation to project portfolio selection through a mean-Gini approach and stochastic dominance, the impact of uncertainty on project parameters. In the process, Monte Carlo simulation is consid- eredinevaluatingtheimpactofparametricuncertaintyonprojectselec- tion. The results show that the influence of uncertainty is significant enough to mislead managers. A more robust selection policy using the mean-Gini approach and Monte Carlo simulation is proposed.

Introduction

Managers select projects by prioritizing some options over others and by excluding options that are not aligned with their company strategy or that may lead to a loss; the choices about which the manager makes decisions are usually treated as a portfolio. Portfolio theory seeks to manage risk in a group of assets to determine a combination that offers the lowest risk and the highest expected return. Such a group is called an optimal portfolio. As with a finan- cial portfolio, portfolio management focuses primarily on select projects to ensure that risks, complexity, potential returns, and resource allocation are aligned to the organization’s strat- egy to provide optimal benefits (Petit 2012). Thus, if a project’s expected return and its associ- ated risk can be estimated, portfolio theory can be used to select the most attractive options. The concept of portfolio selection, which was introduced by the seminal work of Markowitz (1952), established the optimal strategy for maximizing return and minimizing the associated variance. When this strategy is followed, the efficient frontier is reached, where for a given variance level, there exists no other portfolio with a greater expected return. Similarly, for a given expected return level, there exists no other portfolio with smaller variance.

CONTACT Guilherme Augusto Barucke Marcondes [email protected] National Institute of Telecommunications, Department of Computing Engineering, Av. João de Camargo, , Santa Rita do Sapucaí -, Brazil. Color versions of one or more figures in the article can be found online at http://www.tandfonline.com/utee. ©  Institute of Industrial & Systems Engineers

34 G. A. BARUCKE MARCONDES ET AL.

Although project portfolios are different from financial portfolios due to the absence of market price as a factor, financial portfolio theory still may be useful for project portfolio selec- tion analysis (Casault et al. 2013). Indeed, the mean variance portfolio (MVP) of Markowitz (1952) has been directly applied to R&D project selection. Sefair and Medaglia (2005) dis- cussed selection based on net present value and variance, considering budget constraints, as well as interdependency among projects and timeframes. Medaglia et al. (2007), in turn, proposed project selection by using MVP considering partially funded projects, interdepen- dencies, and resource constraints. However, Feldstein (1969) showed that the mean variance framework provides reliable results only when the returns are normally distributed or when the utility function is risk-averse quadratic. In fact, such assumptions are only very narrowly applicable for real-world problems such as project portfolio selection, as was pointed out by Better and Glover (2006).

Though Markowitz’s work has been explored extensively, many authors have contributed to it and modified it in important ways. An interesting approach to portfolio analysis was discussed by Yitzhaki (1982), who used the mean and the Gini mean difference to describe the distribution of return. Yitzhaki (1982) focused on stochastic dominance (SD) criteria, which is less restrictive than MVP. Based on this discussion, Shalit and Yitzhaki (1984) introduced the mean-Gini (MG) approach applied to portfolio analysis, which Ringuest et al. (2004) adapted to R&D portfolio selection. Ringuest et al. (2004) have pointed out at least four advantages in applying the MG approach to project portfolio selection: (i) the approach is simple and intuitive; (ii) it allows for the construction of portfolios that are efficient by the criteria of stochastic dominance; (iii) it is applicable to risk-averse decision makers; and (iv) it requires no explicit knowledge of the decision maker’s utility function.

In solving the MG portfolio, Ringuest et al. (2004) relied on the assumption that the returns distribution is known, so that the authors do not have to incorporate parameter estimation error into the project modeling. In such cases, the managers must choose the projects that yield exactly the value obtained from a mathematical model. In real cases, however, project parameters that are unknown either cannot be fully specified or may change during the project’s execution.

Many approaches for project portfolio selection under uncertainty have been proposed in the literature. Hildebrandt and Knoke (2011) reviewed some methods for investment deci- sions under uncertainty for applications in projects for forest management such as methods based in expected utility, option price models, and robust optimization. Hassanzadeh et al. (2014) analyzed the use of robust optimization in project portfolio selection, concluding that robust solutions show a high degree of feasibility, as well as average and worst-case perfor- mances that are comparable to those of the nominal solutions. Fuzzy sets have also been used to deal with uncertainty in project portfolio selection (Bhattacharyya et al. 2011; Perez and Gomez 2014), and linear programming solvability has important applications to real prob- lems (Mansini et al. 2014). Dutra et al. (2014) proposed an economic-probabilistic model for project selection and prioritization, and Ghapanchi et al. (2012) used an approach based on data envelopment analysis. With regard to multicriteria decision analysis methods in project selection with uncertainty, Huang et al. (2008) have applied fuzzy analytic hierarchy process (AHP) for R&D project selection sponsored by government. Oztaysi (2015) applied similar fuzzy AHP for information systems project selection. The technique for order of preference by similarity to ideal solution (TOPSIS) was applied by Tan et al. (2010) to assist contractors in project bidding. TOPSIS has been applied by Collan and Luukka (2014) for ranking R&D projects to help in decision making. AHP and TOPSIS may also be used together, as has been done by Amiri (2010), who used AHP to analyze the structure of the project selection problem

THE ENGINEERING ECONOMIST 35

and to determine the weights of the criteria, as well as the fuzzy TOPSIS method for obtaining a final ranking. Finally, simulation approaches are also used for project analysis (Better and Glover 2006; Shakhsi-Niaei et al. 2011).

The Project Management Body of Knowledge (PMBOK) Guide (Project Management Institute [PMI] 2013) suggests that project parameters may be estimated by expert judgment or interviewing techniques, but such an approach may lead to discrepancies among ana- lysts, resulting in parameter uncertainty. Once the project parameter has been established, Monte Carlo simulation (MCS) may be used for quantitative risk analysis. However, in cases where the project parameters fluctuate, the expected portfolio result may be strongly affected. Parameter uncertainty, indeed, plays an important role in portfolio selection, so such a draw- back may cause a quantitative estimation error analysis over different scenarios to become necessary.

Jobson and Korkie (1980), Michaud (1989), Best and Grauer (1991), and Chopra and Ziemba (1993) have all discussed the impact of estimation error on optimal portfolio choice, which is mainly explained by difficulties in estimating expected return. Along the same lines, Lim et al. (2011) showed that this drawback is also observed when other coherent risk met- rics are used. However, although project portfolio selection has been widely discussed in the literature, few works discuss the impact of uncertainty in project portfolio selection, where the efficient frontier is discrete.

In discussing such pitfalls, this article analyzes a portfolio selection policy based on an integrated simulation framework composed of both quantitative risk analysis and the MG approach, so that estimation error in R&D project portfolio selection can be more effec- tively dealt with. To do so, rather than just using a point estimate for uncertain parameters, we incorporate some uncertainty about these quantities into the model. Thus, we can simu- late the MG algorithm proposed by Ringuest et al. (2004) based on quantitative risk analysis of each project. The simulation results are then used to estimate the efficient frontier confi- dence interval and to perform a stochastic dominance analysis, helping managers in decision making.

Simulation-based approaches are normally used when the mathematical model is very dif- ficult (or impossible) to track. Better and Glover (2006) discussed such issues in project port- folio selection, where different risk types may be considered in prospective decisions; such risks include a project’s probability of success, uncertainties in sales and cost projections, and estimation errors in projected investment requirements, among others. Furthermore, the dis- crete nature of project portfolio choice yields a discrete efficient frontier, hindering the esti- mation of such a frontier. Thus, the contribution of this article relies on tracking efficient fron- tier portfolios considering the estimation error of model parameters using the MG portfolio approach.

Although restrictions are important in the selection of project portfolios, none are con- sidered here. The focus in this article is on financial portfolio return and risk analysis. The proposed policy uses the static selection approach, as classified by Eilat et al. (2006) and Urli and Terrien (2010), which consider only candidate projects (projects waiting to start) and not active projects (running projects). In so doing, we discuss how the uncertainty in project parameters may affect the SD criteria.

The remainder of this article is organized as follows: the following section defines the con- cept of risk and uncertainty adopted in this study for project portfolio selection. The next sec- tion presents the mean-Gini portfolio model for project selection. The next section discusses how estimation error affects portfolio choice. The following section discusses quantitative risk analysis for portfolio selection. A portfolio selection policy is proposed next and numerical

36 G. A. BARUCKE MARCONDES ET AL.

results based on a real set of projects are presented. Finally, the conclusions are stated in the last section.

Risk and uncertainty in project portfolio selection

When many projects are available, managers must select the ones that promise the highest possible return and minimum risk, considering an organization’s indifference curves and the projects that align most closely to strategic goals. It is critical to minimize, as much as possible, the effect of the uncertainties inherent in forecasting.

Sanderson (2012) argues that risk and uncertainty are closely related and that there is a tendency in project literature to use these terms interchangeably so that the concept of uncertainty is either treated the same as risk or is simply ignored. The PMBOK (PMI 2013) defines risk as “an uncertain event or condition that, if it occurs, has a positive or a nega- tive effect on at least one project objective, such as time, cost, scope, and quality,” (p. 310). The Guide acknowledges risk as “the effect of uncertainty on projects and organizational objectives,” (p. 310). Thus, uncertainty and risk are better described as cause and conse- quence, so that risk is considered as one of the implications of uncertainty (Perminova et al. 2008).

In an effort to clarify the key distinctions between risk and uncertainty, Knight (1964), who was one of the first to consider the difference, referred to uncertainty as an event for which one cannot specify some probability of occurrence, whereas for risk, one is able to do so. Bannerman (2008) assumed risk as foreseeable threats (“known unknowns”) and uncer- tainty as unforeseeable threats (“unknown unknowns”). Similarly, Cleden (2009) defined uncertainty as the intangible measure of “we don’t know” (again, unknown unknowns) and risk as the statement of what may arise from that lack of knowledge (as above, known unknowns).

Although authors may differ in defining such concepts, it is imperative to distinguish them in order to understand the influence of uncertainty and risk individually in project perfor- mance (Perminova et al. 2008; Petit 2012).

General definitions of uncertainty and risk broadly encompass project portfolio manage- ment, in which one is attempting to manage risk while projects are being executed. Here, instead, we are considering project portfolio selection. Thus, projects are previously identified and assessed to determine the degree of alignment with and contribution to one or more of an organization’s strategic objectives. Afterwards, based on all information available, projects must be selected to attain a certain level of return at a certain level of risk. Evaluating risk in projects includes several processes, including quantitative risk analysis. PMI (2013) defines such an approach as the process of numerically evaluating the effect of all identified risks and uncertainties over the project objectives. Based on the brief discussion above, in this article we consider the definitions for uncertainty and risk similarly to Jaafari (2001):! Uncertainty: the probability that the project parameters will not reach their expected

(planned) values,! Risk: the exposure to loss/gain based on some statistical measure. The definitions above are incorporated in this article in the portfolio analysis and the Gini

coefficient as a risk measure is explored. The Gini coefficient is a statistical dispersion measurement proposed by Gini (1912) to

represent the income distribution of people who live in a country or region. Aside from its primary objective, it has also been applied in several other kinds of dispersion measurement. Its application to portfolio analysis is discussed in the next section.

THE ENGINEERING ECONOMIST 37

Mean-Gini and stochastic dominance for project portfolio selection

Shalit and Yitzhaki (1984, 1989, 2005) introduced the application of MG in risk and portfolio analysis where such an approach appears as an alternative to traditional MVP. They showed that this approach is less restrictive than traditional MVP, proving that the MG approach does not depend either on quadratic utility or on the normality distribution assumption of returns. Notwithstanding that, the application of MG keeps the simplicity of the proposed approach by Markowitz (1952). Ringuest et al. (2004) and Graves and Ringuest (2009) applied the MG approach to risky R&D project portfolio selection. As the authors argued, the method is simple and intuitive and requires few parameters, so an efficient portfolio frontier can be constructed.

A portfolio’s expected return is the sum of all expected returns of the selected projects. If xi ∈ {0, 1} represents the decision made in rejecting or selecting a project i and Ri its associated expected return, a portfolio’s expected return RP may be defined, for N projects to be selected, by Equation (1):

RP = N!

i=1 xi · Ri. (1)

The Gini statistic is defined as the expected distance between two realizations of the same random variable. Shalit and Yitzhaki (1984) proposed a practical representation for computa- tional purposes as twice the covariance between the return RP and its cumulative distribution function F (RP), as shown in Equation (2):

!P = 2 · cov [RP, F (RP)] . (2)

In constructing an efficient frontier, Ringuest et al. (2004) proposed a procedure based on a branch and bound algorithm. Such an approach requires basic data on the return distri- bution from each project. By using the mean and Gini coefficient for portfolios, the branch and bound approach branches out to all possible combinations of available projects, starting with the selection of all projects available. Upper and lower bounds are computed, and the portfolio tree is “pruned.” In the MG sense, a portfolio is dominated if any other portfolio has either a higher mean return with the same or a lower Gini or a lower Gini with the same or a higher mean return. Based on such dominance assumptions, for each branch on the branch and bound approach, it can be determined which branches can be pruned from the analysis.

Ringuest et al. (2004) and Graves and Ringuest (2009) also proposed using SD as a criterion to evaluate candidate portfolios. Indeed, SD criteria allow consideration of the entire prob- ability distribution, indicating when a random variable ranks higher than others. Thus, this approach has been gaining attention in risk analysis (Levy 1992; Graves and Ringuest 2009). In portfolio selection, the first- and second-order SD are usually considered. Using a first- order SD (FSD), it is possible to identify when a random variable is stochastically larger than another, whereas the second-order SD (SSD) enables the stochastically less volatile variable to be identified. Over the remaining portfolios (dominants) included in the efficient frontier, SD criteria can then be applied to choose which portfolios will “survive.” As presented by Ringuest et al. (2004), portfolio i stochastically dominates portfolio j (SSD) if, for all z over RP, one has

" z

−∞

# F

$ RPj

% − F

& RPi

'( dRP ! 0 (3)

and the remaining portfolios after SD analysis are the candidate portfolios to be chosen.

38 G. A. BARUCKE MARCONDES ET AL.

Table . Set of five projects (Ringuest et al. ).

Project High return Low return P (high return)

A   . B , − . C   . D  − . E  − .

In solving the MG portfolio as described above, two points are worth mentioning. First, the SD analysis is made over the return distribution. If other selection criteria are also con- sidered, such criteria must be modeled as random variables and multivariate SD analysis may be performed (see O’Brien and Scarsini [1991] for details). Furthermore, return distributions are assumed to be known. However, this is usually not the case for real-world projects. Thus, in what follows, we discuss how the estimation error might affect decision making.

The role of parameter uncertainty

The MG approach discussed in the previous section depends on the return distribution in order to compute two parameters: the project’s portfolio return and the Gini coefficient. If return distributions are known, these two parameters can be computed, so the methodology discussed in the previous section can be directly applied. In practice, however, return distribu- tions that are unknown are estimated using available information at the moment of assessment about the projects under analysis. Such an approach generally results in uncertainty about the return distribution parameters.

Uncertainty in parameter estimation may lead to poor out-of-sample portfolio perfor- mance. Jobson and Korkie (1980), Michaud (1989), Best and Grauer (1991), and Chopra and Ziemba (1993) have all discussed the impact of estimation error on optimal portfolio choice, showing that it plays an important role in portfolio selection. In discussing how the estimation error may affect decision making in project portfolio selection, we use a simple five-project example borrowed from Ringuest et al. (2004), so that the impact of uncertainty over project parameters can be easily shown. The data set presented in Table 1 exemplifies the features of the proposed application. Also in Table 1, the second and third columns present the mone- tary return if the project is successful (high return) or unsuccessful (low return), respectively. The fourth column shows the probability of a high return. If no uncertainty over parameters (high return, low return, and high return probability) is being considered in project param- eters, the efficient frontier is defined by the set of portfolios: ABCE, ACE, BCE, AC, CE, and C.1 The mean and Gini coefficient of each portfolio are presented from the second to the fourth columns of Table 2, using the “original” results from Ringuest et al. (2004).

In order to evaluate the impact of estimation error on the efficient frontier, an efficient frontier analysis may be employed by considering some uncertainty in project parameters, for instance, considering first ± 10% uncertainty over the probability of success of project B. Such uncertainty has an impact on the portfolio return and Gini coefficient, as it may also affect the efficient frontier. This is true because the analyst may observe any value between 0.45 or 0.55, instead of the parameter 0.50. In exercising such uncertainty, we compute the

 See Ringuest et al. () for details. We kept the same parameter values proposed by Ringuest et al. (), so the comparison among“original”resultsandresultswithuncertaintycouldbeperformed.Furthermore,thenotationassumedfollowsRinguest et al. (). For example, in the portfolio ABCDE, all projects are selected; in the portfolio ABCD, all projects except project E are selected, and so on.

THE ENGINEERING ECONOMIST 39

Table . Mean and Gini—Project B parameter uncertain.

Project B success probability .—“original” . .

Portfolio Mean Gini Efficient Mean Gini Efficient Mean Gini Efficient

ABCE . . Yes . . Yes . . Yes BCE . . Yes . . Yes . . No ACE . . Yes . . Yes . . Yes AC . . Yes . . Yes . . Yes CE . . Yes . . Yes . . Yes C . . Yes . . Yes . . Yes

efficient frontier for the extreme points 0.45 and 0.55, comparing the results to the “original” results from Ringuest et al. (2004), which are shown in the fifth to the tenth columns of Table 2.

Note that in Table 2, with a probability of success of 0.55 in project B, the efficient frontier has changed and is composed of the following set of portfolios: ABCE, ACE, AC, CE, and C. Different from the original case, portfolio BCE no longer belongs to the efficient frontier due to its new Gini coefficient. At the original efficient frontier of Table 2, instead, BCE is not dominated by any other portfolio, but its Gini coefficient is very close (a little bit lower) to portfolio ABCE’s Gini. When the probability of success varies from 0.50 to 0.55 in project B, portfolio BCE’s Gini increases from 439.39 to 448.21, and its mean has changed from 625.0 to 687.5. Similarly, portfolio ABCE is also impacted by uncertainty in project B (its mean has changed from 875.0 to 937.5 and its Gini has increased from 441.67 to 445.83). Thus, the observed portfolio BCE becomes dominated by ABCE, so that it no longer lies in the efficient frontier and it can be seen that a small perturbation in the project parameter may lead an analyst to a wrong decision.

With a probability of success in project B at 0.45, on the other hand, the efficient portfolios do not change. However, the observed efficient frontier is different, as the mean and Gini values have changed. Indeed, the means of all portfolios containing project B are reduced by 62.5, and the reduction of the Gini coefficient ranges from 2.02 to 20.83. This case is illustrated in Figure 1.

Similar behavior may be observed if ± 10% uncertainty is considered for the probability of success of project D. Again, we exercise such uncertainty by computing the efficient fron- tier for the extreme points 0.45 and 0.55. The results are shown in Table 3. Note from this table that, when the success probability of project D is 0.55, portfolios ABCDE and ACDE become efficient, so that the efficient set has changed to ABCDE, ABCE, ACDE, ACE, BCE,

Figure . Efficient frontier—Project B success probability variation.

40 G. A. BARUCKE MARCONDES ET AL.

Table . Mean and Gini—Project D parameter uncertainty.

Project D success probability .—“original” . .

Portfolio Mean Gini Efficient Mean Gini Efficient Mean Gini Efficient

ABCDE . . No . . No . . Yes ABCE . . Yes . . Yes . . Yes ACDE . . No . . No . . Yes BCE . . Yes . . Yes . . Yes ACE . . Yes . . Yes . . Yes AC . . Yes . . Yes . . Yes CE . . Yes . . Yes . . Yes C . . Yes  . Yes . . Yes

Figure . Efficient frontier—Project D success probability variation.

AC, CE, and C. In the original result, however, portfolio ABCDE was dominated by portfolio ABCE, and portfolio ACDE was dominated by ACE. Similar to the previous case, with the probability of success in project D at 0.45, the efficient portfolios do not change, but there is a different efficient frontier (see Figure 2). Interestingly, note that project D is not included in any portfolio in the set of efficient portfolios at the “original” efficient frontier of Table 3. The uncertainty in project D parameter, however, may induce the analyst to observe project D as the efficient choice in certain cases, which is a misleading result.

This section closes with a very interesting simulation. We now consider different levels of uncertainty over the probability of success of the five projects all together. In this case, MCS is used to gather insight on the impact of uncertainty on R&D efficient frontier analysis. The goal is still to analyze how parameter uncertainty might affect decision making. Thus, 2,000 MCS replications were performed, considering different uncertainty levels (± 1%, ± 2%, and so on) to construct the efficient frontier. A natural choice for modeling such uncertainty is a probability distribution with high density in the estimated parameter and lower density in the borders of the uncertainty interval, so that large misspecification is avoided. Hence, in this sensitivity analysis we used the triangular distribution to model the uncertainty in the probability of success of the five projects.2 The results were then compared to the original efficient frontier obtained with the data of Table 1. The analysis is presented in Figure 3, which shows the proportion of success in selecting the original efficient portfolios; that is, the ones that are efficient when the success probability is 0.50 for all projects.

 Probability models for uncertainty outcomes, such as triangular distribution, are discussed next, in Section .

THE ENGINEERING ECONOMIST 41

Figure . Results for MCS—Uncertainty in all projects combined.

Note in Figure 3 that the original efficient frontier is correctly identified in at most 52% of the cases when MCS is used and an uncertainty of ± 1% is considered. That is, if all projects have an uncertainty of ± 1% over their probability of success, the original efficient set of port- folios would be identified in at most 52% of the cases. This is a very poor result in analyz- ing such project portfolios. Moreover, the situation gets even worse if the level of parameter uncertainty increases. Note that, for an uncertainty of ± 10%, success in selecting the origi- nal efficient frontier decreases to at most 40%. These results suggest that the uncertainty in discrete portfolio selection may be even worse than in continuous portfolio selection.

A case where the level of uncertainty is ± 10% is illustrated in Figure 4 for 2,000 MCS replications. In the figure, many “clouds” of points can be seen. Each cloud represents port- folio information resulting from MCS considering the uncertainty of ± 10% over the proba- bility of success of all five projects. Note that in Figure 4 similar conclusions are reached. For instance, portfolios AC and CE are efficient based on the original data. However, depending on the uncertainty level on project parameters, portfolio CE is dominated by AC. Similarly, the original efficient portfolio BCE may be dominated by ACE or ABCE. On the other hand, the non-original efficient portfolios, ACDE and ABCDE, are efficient.

Figure . Efficient frontier—Uncertainty in all projects combined.

42 G. A. BARUCKE MARCONDES ET AL.

Based on these results, it can be concluded that uncertainty has an important influence on the efficient frontier and the efficient set of portfolio results, yielding a risk in portfolio selection. Although the impact of estimation error on portfolio selection has already been observed by Jobson and Korkie (1980), Michaud (1989), Best and Grauer (1991), and Chopra and Ziemba (1993), the cases explored above translate such observations to project portfolio selection in which the efficient frontier is discrete. Furthermore, depending on the number of projects available and the level of uncertainty, the analyst may by choosing the project portfo- lio badly, take unnecessary risk when a better option might be available. Certainly, the greater the number of projects, the bigger will be influence of uncertainty over the results. Thus, selecting portfolios based on a quantitative risk analysis that takes parameter uncertainty into account may lead to a more robust portfolio choice, as will be discussed next.

Quantitative risk analysis for portfolio selection

Quantitative risk analysis may be defined as the process of numerically evaluating the effect of all identified risks and uncertainties over the project objectives (PMI 2013). In general, such analysis uses a simulation approach, such as MCS, and is performed prior to project selection so that the analyst may be aware of the inherent risks of the portfolio choices.

When risk in portfolio analysis is quantitatively evaluated by simulation, probability distri- butions are used to model the parameters of the projects. Two continuous distributions widely used to model uncertainty in returns are the triangular distribution and beta distribution. The former uses a three-point estimate defining best-case, most likely, and worst-case outcomes. The latter is defined by two parameters of shape that may be adjusted so that the distribution is compatible with typical returns (see PMI [2013] for discussion). On one hand, a triangular distribution is easy to define, and its parameters are very intuitive. On the other hand, inter- preting the parameters of beta distribution is more challenging (Dorp and Kotz 2002; Johnson 1997; Stein and Keblis 2009; Yang 2005). Other distributions, such as normal or lognormal, may also be used. These distributions share the feature that the most likely outcome has a high density value, but the density decays when approaching best-case and worst-case outcomes. In cases where no obvious outcome is specified, uniform distribution may be used.

Based on the discussion above, we adopted the triangular distribution to model the project returns for their simplicity and intuitiveness. In this approach, three estimates for the project parameters (time, effort, and cost) are considered. Such estimates may be obtained by expert judgment or interviewing techniques. Thus, considering best-case, most likely, and worst- case outcomes estimates for project return, the triangular probability distribution can be set such that some simulations can be performed to evaluate project performance and risk (PMI 2013).

As discussed by Kitchenham and Linkman (1997) and Stamelos and Angelis (2001), any estimation method may lead to some uncertainty in establishing project return distributions, so the risk analysis may be mistaken. In fact, Atkinson et al. (2006) mentioned that one of the causes of uncertainty about estimates is bias that may be exhibited by estimators. In dealing with this drawback, a range of the parameters of project return distribution may be used. For instance, in establishing the worst-case return, an expected value can be set, as well as its lower and upper bounds. It can also be assumed that all values in the range of the lower and upper bounds have the same probability of occurrence.3 The same procedure may be applied to most likely and best-case return estimates. This approach in establishing return distribution yields what may be called an uncertain triangular distribution. Figure 5 illustrates, with a solid black line, a triangular distribution for a project whose worst-case, most likely,

 Note, however, that any other probability distribution might be assumed.

THE ENGINEERING ECONOMIST 43

Figure . Illustration of the uncertain triangular distribution.

and best-case returns estimates are −$100, $500, and $600, respectively. However, considering estimation error, it can be assumed that the worst-case estimate might range from −$150 to −$50, whereas the most likely and best-case estimates might range from $450 to $550 and $550 to $650, respectively, yielding the uncertain triangular distribution (gray shaded area).

Note that this approach enables us to evaluate the quantitative risk while dealing with uncertain parameters. Thus, in performing the portfolio selection considering estimation error, the uncertain triangular distribution is used to analyze the MG efficient frontier by means of MCS. Considering the discussion above, we go on to analyze a selection policy for project portfolios by considering uncertainty in parameter estimation so that the portfolios chosen are more likely to be at the efficient frontier.

A portfolio selection policy considering uncertainty

The selection policy proposed in this article is performed in three stages: (i) identifying the possible efficient portfolios; (ii) selecting portfolios with greater probability of being dom- inant; and (iii) applying the stochastic dominance criteria. To accomplish such a selection policy, the branch and bound algorithm discussed in the section Mean-Gini and stochastic dominance for project portfolio selection as well as the MCS approach are considered.

The MCS approach is used in the first stage at two points. In the inner loops, MCS is used to estimate the expected return and Gini coefficient of the portfolio under analysis. In an outer loop, MCS is considered in order to obtain expected values (for point estimate analysis) and confidence levels (for estimation error analysis) for the estimated expected return and the Gini coefficient. For each outer MCS trial, the branch and bound algorithm is used to obtain the trial’s efficient frontier. Portfolios that lie at least once in the efficient frontier are considered to be candidates.

Once the simulation is finished, the second stage may be performed by obtaining the mean values or the confidence interval (CI) for the portfolio return and the Gini coefficient. When working with expected values, the MG dominance evaluation is done using the criteria pre- sented in the section Mean-Gini and stochastic dominance for project portfolio selection. Applying CI to analyze the efficient portfolio candidates, portfolio i dominates portfolio j if

RCILowi ≥ R CI Up j

(4a)

44 G. A. BARUCKE MARCONDES ET AL.

!CIUpi ≤ !CILow j , (4b)

where RCILowi and R CI Up j

represent the lower and upper bound for the return CI of portfolios i and j, respectively. Likewise, !CIUpi and !

CI Low j

represent the lower and upper bound for the Gini CI of portfolios i and j. Note that, as in point estimate dominance, if either (4a) or (4b) is satisfied by equality, the other one must be satisfied by inequality.

Next, the analysis considers returns and Gini CI overlaps, respectively. If, for a pair of port- folios, some overlap in CI (return and/or Gini) occurs, the one with that higher probability (that is, which occurred more times during the simulation) dominates the other one. The port- folios that present higher probabilities of being dominant remain candidates, whereas the rest are discharged.

Finally, in the third step, the remaining portfolios are tested under SD (in this case, we applied SSD) criteria. Again, only the remaining stochastic dominant portfolios are candidates to be executed.

The experiment described above may be summarized by the following Algorithms 1 and 2.

Algorithm 1 Start project parameters loop L times

Generate B random values for each triangular distribution; Apply the branch and bound algorithm discussed in the section Mean-Gini and

stochastic dominance for project portfolio selection; Store portfolios at efficient frontier as candidates portfolios.

end loop Eliminate dominated portfolios; Apply the stochastic dominance criteria.

Algorithm 1 above considers, indeed, only point estimates on the triangular distribution of the portfolios. Recall, however, that the goal of this article is to discuss the impact of estima- tion error in portfolio selection policy. Thus, a slight change may be made in order to consider estimation uncertainty in triangular distribution parameters, as discussed in the previous sec- tion. This change leads to Algorithm 2:

Algorithm 2 Start project parameters loop L times

Sample triangular distribution from uncertain triangular distribution for each project; Generate B random values for each triangular distribution; Apply the branch and bound algorithm discussed in the section Mean-Gini and

stochastic dominance for project portfolio selection; Store portfolios at efficient frontier as candidates portfolios.

end loop Compute the confidence interval for candidate portfolios; Eliminate portfolios that might be dominated through CI criteria; Apply the stochastic dominance criteria.

THE ENGINEERING ECONOMIST 45

By running Algorithm 1 and different uncertainty scenarios for Algorithm 2, the proposed selection analysis can be performed. Notice that the literature suggests that using more than one selection method tends to result in better performance in the portfolio analysis (Cooper et al. 2001). Thus, the selected portfolios must be the ones chosen in all different runnings (including both algorithms). When applying this policy, the choice is more robust, because the selected portfolios are dominant for different uncertain scenarios evaluated (including point estimate and different levels of uncertainty).

Portfolio selection considering estimation error—Monte Carlo experiments

In this section, some numerical experiments are presented following the discussion of the pre- vious sections. The goal of the simulations discussed here is to apply and analyze the selection policy presented in the previous section in order to obtain candidate portfolios, taking esti- mation uncertainties into account.

Algorithms 1 and 2 were performed by using data from 10 software development projects from an anonymous company4 to be developed by a project service provider that has operated in Brazil for more than 20 years.

In order to estimate a triangular distribution for project return, the estimation process for each project involved three software engineers with more than 10 years of experience on software development playing the role of software architect for Web-based systems; all have experience in object-oriented software system development and modeling. They also have experience in databases. For each project, worst-case, most likely, and best-case scenarios were analyzed individually by each software engineer, considering workforce. After individual esti- mation, a meeting with the software engineers was held to arrive at a consensus estimation yielding the triangular distribution for the projects. Based on the consensus estimation for workforce, the cost for each project was obtained, and the price was computed by adding up the margin value.

Note that this estimation method, in practice, leads to deterministic values for the trian- gular distribution parameters. Such standard practice is used here as a reference. However, in this study, the individual estimates were considered as interval estimations. That is, for each estimate (worst-case, most likely, and best-case), the lower and upper bounds were considered as interval estimates.

Table 4 presents the range of expected return for each project. In this table, LB and UB refer to the lower and upper bounds of parameter estimation, respectively (the bounds observed by the differences among the software engineers’ individual estimations). The table also shows the consensus on triangular distribution for the expected return.

Algorithm 1 was run using the consensus values presented in Table 4. In Algorithm 2, which enables the analysis of estimation errors, three scenarios were drawn based on the information in Table 4 in order to gather insight about the estimation error effect in port- folio selection. The first and second scenarios were drawn from the exercise performed in the section The role of parameter uncertainty (Figure 3). The goal in these scenarios was to analyze how uncertainty levels regarding project parameters affect the portfolio choice. The third scenario considers LB and UB as the lower and upper limits of parameters. The aim of scenario 3 was to analyze the nuisance data in the estimation method. In other words, we

 Although the data set was real, the name of the company is omitted under contractual restriction.

46 G. A. BARUCKE MARCONDES ET AL.

Table . Expected return ($).

Project Scenario LB UB Consensus Project Scenario LB UB Consensus

Best-case , , , Best-case , , , A Most likely , , , F Most likely , , ,

Worst-case −, −, −, Worst-case −, −, −, Best-case , , , Best-case , , ,

B Most likely , , , G Most likely , , , Worst-case −, − −, Worst-case −,  −, Best-case , , , Best-case , , ,

C Most likely −, , , H Most likely , , , Worst-case −, −, −, Worst-case −, −, −, Best-case , , , Best-case , , ,

D Most likely , , , I Most likely , , , Worst-case −, , −, Worst-case −, , −, Best-case , , , Best-case , , ,

E Most likely , , , J Most likely , , , Worst-case −, , −, Worst-case − , , −,

wanted to consider the difference between individual project analysis and consensus analy- sis. As in the section The role of parameter uncertainty, a triangular distribution was used to model the uncertainty in the projects’ parameters so that large misspecification could be avoided. Thus, in analyzing portfolio selection with estimation error, the following scenarios were considered:

1. Scenario 1: The parameters for the portfolio return triangular distribution were ran- domly chosen between the ranges LB and UB, considering LB as the consensus value (as presented in Table 4) less 2% and UB as the consensus value plus 2%;

2. Scenario 2: The parameters for the portfolio return triangular distribution were ran- domly chosen between the ranges LB and UB, considering LB as the consensus value (as presented in Table 4) less 5% and UB as the consensus value plus 5%;

3. Scenario 3: The parameters for the portfolio return triangular distribution were ran- domly chosen between the ranges LB and UB, as presented in Table 4.

Numerical results

In the following, the numerical results are presented based on the methodology described above. In the first case analyzed, the 10 project data in Table 4 are considered in Algorithm 1. Next, the estimation error is taken into the analysis by performing Algorithm 2 in the three different scenarios described in the previous section. For all cases, L = 2,000 and B = 2,000, and 95% confidence for CI evaluations.

Point estimate analysis In the point estimate analysis, Algorithm 1 used the consensus values shown in Table 4. In this case, the MCS stage led to 167 portfolios as possibly efficient. That is, when running the MCS, 167 portfolios lay at least once in the efficient frontier.

In the second stage, these 167 portfolios were then analyzed with the expected values. Thus, the dominance analysis was done considering the portfolio return and Gini coefficients’ expected values. If a portfolio i has an expected return greater than a portfolio j’s expected return, and the expected Gini of portfolio i is lower than the expected Gini of portfolio j, then portfolio i dominates portfolio j. In this case, portfolio i remained as a candidate, and portfolio j was discharged from the efficient frontier. In performing this analysis, the number of candidate portfolios was reduced to 85.

Finally, in the third stage, when SD criteria were applied, 60 candidate portfolios remained. In this analysis, and based on the selection policy considered in this article, only the stochastic

THE ENGINEERING ECONOMIST 47

Table . CI overlap evaluation—Example.

Portfolio Return CI Gini CI Probability (%)

i 261, 413.3 261, 477.3 27, 274.7 27, 307.3 95.4 j 259, 846.3 259, 986.4 27, 288.7 27, 389.2 7.0

dominant portfolios remained as candidates to be selected. It should be recalled that the results of point estimate were used as a reference for the following estimation error analysis.

Estimation error analysis To analyze the impact of estimation error on portfolio selection, Algorithm 2 was consid- ered based on the uncertain triangular distribution discussed in the section Quantitative risk analysis for portfolio selection, and on the three different scenarios described above.

In scenarios 1 and 2, the point estimate parameters of the triangular distributions used in Algorithm 1 were replaced by the uncertain triangular distribution defined by an error over consensus value. In scenario 1, the error considered was ± 2%, whereas in scenario 2, the error was ± 5%.

When scenario 1 was performed, the MCS stage led to 172 portfolios as possibly efficient. In the second stage, the dominance analysis was done considering portfolio return and Gini coefficients CI. According to Equations (4a) and (4b), if portfolio i has a return CI greater than the return CI of portfolio j, and the Gini CI of portfolio i is lower than the Gini CI of portfolio j, then portfolio i dominates portfolio j. Thus, portfolio i remains a candidate and portfolio j is discharged from the set of candidates. If, however, some overlap occurs, the decision to keep the portfolio in the efficient frontier is based on the probability of occurrence. That is, portfolio i tends to dominate portfolio j if its probability of occurrence is greater. Table 5 presents an example of this evaluation. In this example, portfolio i remains as a candidate, whereas portfolio j does not. In performing such analysis, the number of candidate portfolios was reduced to 70. Finally, in the third stage, SD criteria were applied, leaving 52 candidate portfolios to be selected. The final efficient frontier for this scenario, plotting the mean values for return and Gini CIs, is presented in Figure 6 and is compared to the point estimate efficient frontier.

Figure . Efficient frontier—Scenario  and point estimate.

48 G. A. BARUCKE MARCONDES ET AL.

Figure . Efficient frontier—Scenario  and point estimate.

In the simulation of scenario 2, the same selection policy was considered. In this case, MCS, CI and SD stages led to 181, 73, and 45 candidate portfolios, respectively. Figure 7 presents the final efficient frontier for this scenario, plotting the mean values for return and Gini CIs, as well as the point estimate efficient frontier.

From the simulation results, the efficient frontier of scenario 1 differs from the point esti- mate approach by about 27%. In numbers, from the 85 portfolios selected as efficient from Algorithm 1, 62 were also selected as efficient when Algorithm 2 was executed with an error of ± 2% in the triangular distribution parameters. As discussed in the section Quantitative risk analysis for portfolio selection, this shows the difficulty of selecting project portfolios consid- ering uncertainty in parameter estimation. However, this difficulty becomes even worse when the SD criteria are considered. Under SD analysis, the difference is about 32%; that is, only 41 out of 60 portfolios identified as stochastic dominant by Algorithm 2 are also dominant at Algorithm 1 under SD criteria. When the error over triangular distribution parameters is about ± 5%, Algorithm 2 produces similar results. In this case, efficient frontier and SD crite- ria analysis differ from the point estimate by about 26% (63 out of 85) and 42% (35 out of 60), respectively.

The results above show how problematic selecting a portfolio with parameter uncertainty is. However, such difficulties may be worsened if one considers the uncertainty of individual estimation. Such uncertainty is represented by LB and UB in Table 4 and will be analyzed next in scenario 3.

In scenario 3, the consensus value that defines the triangular distribution used in Algo- rithm 1 is replaced by the uncertain triangular distribution defined by the LB and UB columns of Table 4. As mentioned earlier, the aim of scenario 3 was the discussion on the difference between individual project analysis and consensus analysis.

When scenario 3 takes place, out of 1,023 possibilities, 211 portfolios lay in the efficient frontier at least once in the MCS stage. After CI analysis, the set of portfolios in the efficient frontier was reduced to 78. Over the remaining portfolios, the SD criteria were applied, leading to 21 candidate portfolios.

It should be noted from the results that the efficient frontier under scenario 3 was quite different from the frontier of the point estimate analysis, where the triangular distribution parameter is given by a group consensus after individual project evaluation. In this scenario,

THE ENGINEERING ECONOMIST 49

Figure . Efficient frontier—Scenario  and point estimate.

out of 85 chosen as efficient in the point estimate analysis, only 31 were also selected as effi- cient. Similarly, out of 60 SD in the point estimate analysis, only six were selected as SD. This represents about 36 and 10% coincidence between individual and consensus estimation anal- ysis, which may be very dubious for portfolio selection. That is, when consensus analysis confronts the uncertainties of individual analysis, the portfolio selection results are very dif- ferent. Figure 8 illustrates the comparison between point estimate analysis and scenario 3, plotting the mean values for return and Gini CIs.

Discussion

The numerical experiments presented in the previous subsection show that portfolio selec- tion may be very difficult with regard to estimation error and uncertainties. Thus, considering different scenarios for estimation error, the results above show how harmful uncertainty may be for portfolio selection. For instance, under scenarios 1 and 2 of estimation error analy- sis, about 60 to 70% of candidate portfolios, the ones that are stochastically dominant, are the same as point estimate analysis yields. The results are even more divergent when indi- vidual estimation (scenario 3) is compared to point estimate analysis. In this case, only 10% coincide.

Based on the policy proposed in the previous subsection, selecting portfolios that are SD efficient in different scenarios seems to be a more robust strategy in project portfolio choice. Then, verifying the results of the numerical experiments above, the candidate portfolios are ABDEFGHIJ, BHIJ, and H, which appear as SD efficients in point estimate analysis and in all three uncertainty scenarios.

Conclusions

This article discusses R&D project selection based on mean-Gini portfolio and stochastic dominance approaches considering estimation uncertainty in project parameters. As project returns are modeled by (uncertain) triangular distribution, the MG approach is considered because it depends neither on quadratic utility nor on the normality distribution assumptions

50 G. A. BARUCKE MARCONDES ET AL.

of returns. It is shown that efficient frontier analysis obtained by a branch and bound approach may be greatly impacted by uncertainty in project parameters. Consequently, the stochastic dominance analysis may also yield different candidate portfolios.

Based on the discussion in this article, numerical simulation analysis considering both triangular and uncertain triangular distribution is carried out. The results show that, as project portfolio selection has a discrete efficient frontier, estimation error plays an important role in its analysis. Depending on the analysis carried out, uncertainty in project parameters leads to misjudgment in selecting project portfolios.

In this sense, different scenarios for uncertainty were simulated (including point estimate, which is the scenario without uncertainty over project parameters). The proposed policy indi- cates the portfolios that are SD efficient in all scenarios as candidates. This choice should be more robust for selecting candidate portfolios, because the dominance analysis is performed by confidence intervals. The bottom line, as different scenarios are simulated and analyzed, is that selecting project portfolios by the proposed approach enables more robustness in candi- date portfolios.

By applying the uncertainty to project expected values, the framework proposed in this article improves the portfolio selection tool, based on MG, dealing with-real world informa- tion. Using triangular distribution for project expected return, as proposed in PMI (2013) for quantitative risk analysis, offers a convenient way for project managers to deal with estimation in projects.

From a managerial point of view, the proposed approach is easy to apply, so analysts can readily simulate and analyze different scenarios. Thus, good project portfolio options may be chosen.

The analysis in this article may be extended in future research by considering resource constraints when building the efficient frontier. Further research may also allow the analysis to be extended for portfolio management, not just portfolio selection.

Funding

Guilherme A. B. Marcondes and Rafael C. Leme would like to thank CNPq (grant 444628/2014-2) for the financial support. Rafael C. Leme also thanks CNPq (grant 307370/2014-3), FAPEMIG (grant PPM- 00185-15) and CAPES (grant 11539/13-5). Carlos E. Sanches da Silva thanks CNPq (grant 310660/2012- 2) and FAPEMIG (grant PPM-00834-15).

Notes on contributors

Guilherme Augusto Barucke Marcondes received B.S. (1991) and M.S. (2005) degrees, all in telecommunications engineering, from the National Institute of Telecommunications, where he is cur- rently a professor. He is seeking a Ph.D. degree in industrial engineering at the Federal University of Itajubá. His areas of interest include project management, optimization, and uncertainty analysis in engineering and management problems.

Rafael Coradi Leme received B.S. (2004), M.S. (2005), and Ph.D. (2008) degrees, all in electrical engineering, from the Federal University of Itajubá, where he is currently a professor at the Institute of Industrial Engineering and Management. His areas of interest include optimization, forecasting, and uncertainty analysis in engineering and management problems.

Marcela da Silveira Leme is seeking a master’s degree in industrial engineering at the Federal Uni- versity of Itajubá, where she is currently a staff member of the Planning Office. The Planning Office is responsible for coordinating strategic and operational planning across the university, providing a range of data, information, analysis, and strategic guidance.

THE ENGINEERING ECONOMIST 51

Carlos Eduardo Sanches da Silva received a B.S. in economics (1989) from the Applied Social Sci- ences College of South of Minas Gerais (FACESM) and a B.S. in mechanical engineering (1990) from the Federal University of Itajubá. He also received his M.D. (1996) and Ph.D. (2001), both in Indus- trial Engineering, from the Federal University of Itajubá and the Federal University of Santa Cata- rina, respectively. He is currently a professor at the Institute of Industrial Engineering and Manage- ment at the Federal University of Itajubá. His areas of interest include project management and product development.

References

Amiri, M.P. (2010) Project selection for oil-fields development by using the AHP and fuzzy TOPSIS methods. Expert Systems with Applications, 37(9), 6218–6224.

Atkinson, R., Crawford, L. and Ward, S. (2006) Fundamental uncertainties in projects and the scope of project management. International Journal of Project Management, 24(8), 687–698.

Bannerman, P.L. (2008) Risk and risk management in software projects: a reassessment. The Journal of Systems and Software, 81(12), 2118–2133.

Best, M.J. and Grauer, R.R. (1991) On the sensitivity of mean-variance-efficient portfolios to changes in asset means: some analytical and computational results. The Review of Financial Studies, 4(2), 315–342.

Better, M. and Glover, F. (2006) Selecting project portfolios by optimizing simulations. The Engineering Economist, 51(2), 81–98.

Bhattacharyya, R., Kumar, P. and Kar, S. (2011) Fuzzy R&D portfolio selection of interdependent projects. Computers & Mathematics with Applications, 62(10), 3857–3870.

Casault, S., Groen, A.J. and Linton, J.D. (2013) Selection of a portfolio of R&D projects. In Handbook on the theory and practice of program evaluation, A.N. Link and N.S. Vonortas, editors. Edward Elgar, Cheltenham, UK.

Chopra, V.K. and Ziemba, W.T. (1993) The effect of errors in means, variances, and covariances on optimal portfolio choice. The Journal of Portfolio Management, 19(2), 6–11.

Cleden, D. (2009) Managing project uncertainty. Gower, Farnham, UK. Collan, M. and Luukka, P. (2014) Evaluating R&D projects as investments by using an overall rank-

ing from four new fuzzy similarity measure-based TOPSIS variants. IEEE Transactions on Fuzzy Systems, 22(3), 302–315.

Cooper, R., Edgett, S. and Kleinschmidt, E. (2001) Portfolio management for new product development: results of an industry practices study. R&D Management, 31(4), 361–380.

Dorp, J.R. and Kotz, S. (2002) A novel extension of the triangular distribution and its parameter esti- mation. Journal of the Royal Statistical Society, 51(1), 63–79.

Dutra, C.C., Ribeiro, J.L.D. and de Carvalho, M.M. (2014) An economic-probabilistic model for project selection and prioritization. International Journal of Project Management, 32(6), 1042–1055.

Eilat, H., Golany, B. and Shtub, A. (2006) Constructing and evaluating balanced portfolios of R&D projects with interactions: a DEA based methodology. European Journal of Operational Research, 172(3), 1018–1039.

Feldstein, M.S. (1969) Mean-variance analysis in the theory of liquidity preference and portfolio selec- tion. The Review of Economic Studies, 36(1), 5–12.

Ghapanchi, A.H., Tavana, M., Khakbaz, M.H. and Low, G. (2012) A methodology for selecting port- folios of projects with interactions and under uncertainty. International Journal of Project Manage- ment, 30(7), 791–803.

Gini, C. (1912) Variabilita e Mutabilita: contributo allo Studio delle distribuzioni e delle relazioni statis- tiche [Variability and mutability: contribution to the study of distributions and relationships statis- tics]. C. Cuppini, Bologna, Italy.

Graves, S.B. and Ringuest, J.L. (2009) Probabilistic dominance criteria for comparing uncertain alter- natives: a tutorial. Omega, 37(2), 346–357.

Hassanzadeh, F., Modarres, M., Nemati, H.R. and Amoako-Gyampah, K. (2014) A robust R&D project portfolio optimization model for pharmaceutical contract research organizations. International Journal of Production Economics, 158, 18–27.

52 G. A. BARUCKE MARCONDES ET AL.

Hildebrandt, P. and Knoke, T. (2011) Investment decisions under uncertainty: a methodological review on forest science studies. Forest Policy and Economics, 13(1), 1–15.

Huang, C.C., Chu, P.Y. and Chiang, Y.H. (2008) A fuzzy AHP application in government-sponsored R&D project selection. Omega, 36(6), 1038–1052.

Jaafari, A. (2001) Management of risks, uncertainties and opportunities on projects: time for a funda- mental shift. International Journal of Project Management, 19(2), 89–101.

Jobson, J.D. and Korkie, B. (1980) Estimation for Markowitz efficient portfolios. Journal of the American Statistical Association, 75, 544–554.

Johnson, D. (1997) The triangular distribution as a proxy for the beta distribution in risk analysis. Jour- nal of the Royal Statistical Society. Series D (The Statistician), 46(3), 387–398.

Kitchenham, B. and Linkman, S. (1997) Estimates, uncertainty, and risk. IEEE Software, 13(3), 69–74. Knight, F.H. (1921/1964) Risk, uncertainty, and profit. Augustus M. Kelleyhn, New York, NY. Levy, H. (1992) Stochastic dominance and expected utility: survey and analysis. Management Science,

38(4), 555–593. Lim, A.E., Shanthikumar, J.G. and Vahn, G.-Y. (2011) Conditional value-at-risk in portfolio optimiza-

tion: coherent but fragile. Operations Research Letters, 39(3), 163–171. Mansini, R., Ogryczak, W. and Speranza, M.G. (2014) Twenty years of linear programming based port-

folio optimization. European Journal of Operational Research, 234(2), 518–535. Markowitz, H. (1952) Portfolio selection. Journal of Finance, 7, 77–91. Medaglia, A.L., Graves, S.B. and Ringuest, J.L. (2007) A multiobjective evolutionary approach for lin-

early constrained project selection under uncertainty. European Journal of Operational Research, 179(3), 869–894.

Michaud, R.O. (1989) The Markowitz optimization enigma: is optimized optimal? Financial Analysts Journal, 45, 31–42.

O’Brien, G.L. and Scarsini, M. (1991) Multivariate stochastic dominance and moments. Mathematics of Operations Research, 16(2), 382–389.

Oztaysi, B. (2015) A group decision making approach using interval type-2 fuzzy AHP for enterprise information systems project selection. Journal of Multiple-Valued Logic and Soft Computing, 24(1), 475–500.

Perez, F. and Gomez, T. (2014) Multiobjective project portfolio selection with fuzzy constraints. Annals of Operations Research, advance online publication. doi:10.1007/s10479-014-1556-z

Perminova, O., Gustafsson, M. and Wikstrom, K. (2008) Defining uncertainty in projects—a new per- spective. International Journal of Project Management, 26(1), 73–79.

Petit, Y. (2012) Project portfolios in dynamic environments: organizing for uncertainty. International Journal of Project Management, 30(5), 539–553.

Project Management Institute. (2013) A guide to the project management body of knowledge. 5th ed. Author, Atlanta, GA.

Ringuest, J.L., Graves, S.B. and Case, R.H. (2004) Mean-Gini analysis in R&D portfolio selection. Euro- pean Journal of Operational Research, 154(1), 157–169.

Sanderson, J. (2012) Risk, uncertainty and governance in megaprojects: a critical discussion of alterna- tive explanations. International Journal of Project Management, 30(4), 432–443.

Sefair, J. and Medaglia, A. (2005) Towards a model for selection and scheduling of risky projects. In 2005 IEEE Systems and Information Engineering Design Symposium, 29 April, Charlottesville, VA.

Shakhsi-Niaei, M., Torabi, S. and Iranmanesh, S. (2011) A comprehensive framework for project selec- tion problem under uncertainty and real-world constraints. Computers & Industrial Engineering, 61(1), 226–237.

Shalit, H. and Yitzhaki, S. (1984) Mean-Gini, portfolio theory, and the pricing of risky assets. The Journal of Finance, 39(5), 1449–1468.

Shalit, H. and Yitzhaki, S. (1989) Evaluating the mean-Gini approach to portfolio selection. The Inter- national Journal of Finance, 1(2), 15–31.

Shalit, H. and Yitzhaki, S. (2005) The mean-Gini efficient portfolio frontier. The Journal of Finance Research, 28(1), 59–75.

Stamelos, I. and Angelis, L. (2001) Managing uncertainty in project cost estimation. Information and Software Technology, 43(13), 759–768.

Stein, W.E. and Keblis, M.F. (2009) A new method to simulate the triangular distribution. Mathematical and Computer Modelling, 49(5–6), 1143–1147.

THE ENGINEERING ECONOMIST 53

Tan, Y., Shen, L., Langston, C. and Liu, Y. (2010) Construction project selection using fuzzy TOPSIS approach. Journal of Modelling in Management, 5(3), 302–315.

Urli, B. and Terrien, F. (2010) Project portfolio selection model, a realistic approach. International Trans- actions in Operational Research, 17(6), 809–826.

Yang, I.-T. (2005) Impact of budget uncertainty on project time-cost tradeoff. IEEE Transactions on Engineering Management, 52(2), 167–174.

Yitzhaki, S. (1982) Stochastic dominance, mean variance, and Gini’s mean difference. The American Economic Review, 72(1), 178–185.

Copyright of Engineering Economist is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.