DEA and prescribing provider

OR Spectrum (2016) 38:743–767 DOI 10.1007/s00291-015-0407-x

REGULAR ARTICLE

An application of value-based DEA to identify the best practices in primary health care

M. C. Gouveia1,2 · L. C. Dias2,3 · C. H. Antunes2,4 · M. A. Mota5 · E. M. Duarte5 · E. M. Tenreiro5

Received: 18 June 2014 / Accepted: 23 June 2015 / Published online: 12 July 2015 © Springer-Verlag Berlin Heidelberg 2015

Abstract In 2005, a new phase for primary health care (PHC) in Portugal began, leading to the voluntary and independent creation of Family Health Units. The impact of this reform is now being witnessed, which justifies the need to evaluate its results as far as best practices in PHC are concerned. This work uses a value-based data envelopment analysis (DEA) method, which combines an additive DEA model with multi-criteria decision analysis concepts, to allow incorporating managerial prefer- ences in the performance assessment of 12 health centres in a Portuguese region. Two models are presented for evaluating access to PHC, which resulted from discussions

B M. C. Gouveia [email protected]

L. C. Dias [email protected]

C. H. Antunes [email protected]

M. A. Mota [email protected]

E. M. Duarte [email protected]

E. M. Tenreiro [email protected]

1 IPC-ISCAC, Quinta Agrícola, Bencanta, 3040-316 Coimbra, Portugal

2 INESC Coimbra, Rua Antero de Quental, 199, 3000-033 Coimbra, Portugal

3 Faculdade de Economia, Universidade de Coimbra, Av. Dias da Silva 165, 3004-512 Coimbra, Portugal

4 DEEC-FCT Universidade de Coimbra, Polo II, 3030-290 Coimbra, Portugal

5 ACES Baixo Mondego I, Rua Luís de Camões, no. 150, 3000-251 Coimbra, Portugal

123

744 M. C. Gouveia et al.

with a group of decision makers. This analysis helped the decision makers in the identification of best practices, sources of inefficiency and gaps relatively to the best practices, thus leading to shape opportunities for improvement.

Keywords Data envelopment analysis · Multi-criteria analysis · Primary health care · Uncertainty

1 Introduction

The primary health care (PHC) system in Portugal is considered of crucial importance intheNational HealthService(NHS). Amajor concernof policymakers is theeffective delivery of health care services (Carter et al. 2012), pursuing objectives of equity, access, efficiency and quality of care.

Several studies of efficiency in provision of PHC exist, especially in the United Kingdom. Most of these studies tend to rely on the use of performance indicators (PIs) to measure health service delivery and quality of care (Donabedian 1980; Baker 1992; van den Hombergh et al. 1998; Campbell et al. 1998; McColl et al. 1998; Proctor and Campbell 1999), but the different PIs are analysed individually. Thanassoulis et al. (1996) and Giuffrida et al. (2000) pointed out the disadvantages of using PIs in performance evaluation and proposed alternative techniques for this purpose, such as data envelopment analysis (DEA). Originally developed by Charnes et al. (1978), DEA is a method for evaluating the relative efficiency of Decision-Making Units (DMUs). It emerged with the objective of evaluating performance in the public sector from the standpoint of productive efficiency, in which there was neither a “market” to pick the most efficient organizational units nor a regulatory model that would guarantee increased efficiency (Sherman and Zhu 2006). Each DMU uses multiple inputs to produce multiple outputs. DEA models seek to determine which of n DMUs form the efficient frontier (or envelopment surface) in the Pareto–Koopmans sense: a DMU lies on the efficient frontier if and only if no other observed DMU improves any input or output without worsening some of other inputs or outputs (Cooper et al. 2004). By identifying the DMUs with the best practices, DEA models construct an empirical production frontier. Since mid-1980s, this methodology is regularly used to evaluate the efficiency of health care services (see Hollingsworth et al. 1999; Sommersguter- Reichmann 2003; Amado and Dyson 2008). Some studies of PHC efficiency use DEA techniques (e.g. Huang and McLaughlin 1989; Giuffrida 1999; Linna et al. 2003; Kirigia et al. 2004; Rosenman and Friesner 2004; Rouse et al. 2011; Thanassoulis et al. 2012), but relatively few studies (Amado and Dyson 2008; Salinas-Jimenez and Smith 1996; Wagner et al. 2003; Nuti et al. 2011; Milliken et al. 2011) include measures for quality of care/service in the efficiency assessment.

Several studies have been undertaken to evaluate efficiency in different areas using DEA, but only a few incorporate managerial preferences in the analysis (see Thanassoulis et al. 2004). The mathematical structure of classical DEA models allows flexibility in the choice of inputs and outputs weights. Therefore, the weights chosen in the solution may be null to important variables, and logical relationships between the variables can be disregarded in a way incompatible with the decision maker’s

123

An application of value-based DEA to identify… 745

(DM) judgment. Hence, the results may not be credible and the DM’s confidence in the methodology may be compromised.

DEA can be combined with multiple criteria decision-making approaches as a means of incorporating managerial preferences in the analysis. For instance, Golany (1988) combined multiple objective linear programming and DEA by developing target-setting models in an interactive way with DMs; Joro et al. (1998) developed a detailed analysis of structural relationships between the two methodologies and Halme et al. (1999) showed that multiple objective linear programming models can be used to incorporate DM’s preferences into DEA when looking for his/her most preferred solution; Mavrotas and Trifillis (2006) proposed a method combining DEA and multi-attribute value theory, in which each alternative uses its most favourable criteria weights and value functions.

This work exploits the links between DEA and multi-criteria decision analysis (MCDA), applied in a real-world situation in which the DMs sought to evaluate DMUs taking into account their managerial preferences. It uses the value-based DEA method (Gouveia et al. 2008), which has proven useful for this type of evaluation in previous studies (Almeida and Dias 2012; Gouveia et al. 2015). This method builds on the addi- tive DEA model with oriented projections (Ali et al. 1995), using concepts developed in the field of MCDA under imprecise information (Athanassopoulos and Podinovski 1997; Dias and Clímaco 2000). Input and output factors are converted into utility functions according to the preference information provided by a group of decision makers (GDM). Additive utility functions are used to aggregate the utilities associ- ated with each alternative, based on multi-attribute utility theory (MAUT) (Keeney and Raiffa 1976). DMUs choose the weights associated with these functions, subject to restrictions reflecting GDM’s preferences. In value-based DEA, the efficiency mea- sure assigned to each DMU has an intuitive meaning: the “min–max regret” (loss of utility) (Gouveia et al. 2008).

The purpose of the present work is to evaluate the efficiency of PHC provided by the 12 units monitored by “Group of Health Centres” (Agrupamento de Centros de Saúde, ACES) Baixo Mondego I, in the central region of Portugal. The study was developed in 2011 for the ACES Baixo Mondego I with 2010 data, concerning 203,657 registered users. Among these users, 19.5 % are over 65-year-old and 22.8 % are less than 25-year-old (PHP 2010). The worth of this work to ACES Baixo Mondego I lies on the ability to use multiple factors in an integrated model and the prospect of identifying best practices, sources of inefficiency and gaps to best practices, as well as contributing to design opportunities for improvement. This approach has revealed to be useful not only for the introduction of corrective measures, but also to inform decisions about the goals of PIs in the future.

In Portugal, there are few studies about performance evaluation of PHC using DEA. Amado and Santos (2009) provide an empirical analysis of efficiency, equity and quality of the PHC in Portugal with the involvement of DMs. They discuss the results with health care professionals and their conviction is that engaging stakeholders in the evaluation process contributes to performance improvement in practice. More recently, Ferreira et al. (2013) also evaluate the performance of PHC providers of a Portuguese region, taking into account both efficiency, equity and quality. In this study, the DMUs are the 22 ACES belonging to Lisbon and Tagus Valley region and the authors show

123

746 M. C. Gouveia et al.

that by establishing a learning network the efficiency of ACES is likely to improve, with a better usage of the available resources while keeping quality and equity.

We suggest two models for performance assessment in PHC. These models are aimed at evaluating, in particular, the accessibility to PHC. From the perspective of theGDM, it is important toinvestigatethepercentageof patients whohaveadesignated general practitioner and use the medical care they are entitled to. The factors included in the models resulted from discussions with the GDM, who are members of the ACES Baixo Mondego I, based on the Performance Plan (PHP 2010) and data from 2010.

This introduction provided the interest and motivation of the study. Section 2 reviews the value-based DEA method (Gouveia et al. 2008) with the modifications to include the super-efficiency concept (Gouveia et al. 2013). Section 3 briefly describes the Portuguese health care system. Section 4 presents the data, the models discussed with the GDM and the elicitation protocols used to obtain the GDM’s preferences. In Sect. 5, the model results are presented and analysed. Section 6 highlights the improvement proposals for the inefficient units and the new developments of the method. Concluding remarks are presented in Sect. 7.

2 The value-based DEA method

The value-based DEA method (Gouveia et al. 2008) is a MAUT-based variant of the additive DEA model with oriented projections (Ali et al. 1995), able to incorporate imprecise information about the DM’s preferences. MAUT is used to evaluate the utility of each alternative, taking into account the multiple criteria. First, the DM focuses his/her attention on one criterion at a time, defining a marginal (partial) utility function for each one. This requires that the original input and output scales are converted into utility scales. There are several techniques for questioning the DM to construct the utility functions compatible with his/her answers (see von Winterfeldt and Edwards 1986; Goodwin and Wright 1998). Then, these functions are aggregated to yield a global utility. For this purpose, the value-based DEA method uses an additive model.

Let uc(aj ) denote the utility of an alternative aj according to the cth criterion (c = 1, . . . , q). The additive utility model aggregates these functions using q scale coefficients w1, . . . , wq as follows (see Keeney and Raiffa 1976):

U(aj ) = q∑

c=1 wcuc(aj ).

The quantity uc(aj ) can be interpreted as a measure of alternative aj’s performance on criterion c, and wc the weight assigned to reflect the importance of the function uc (c = 1, . . . , q). Because utility functions are unique up to positive affine transformations, it is usually assumed that both the global utility functions and marginal utility functions are scaled to lie between 0 and 1. The scale coefficients (weights) are also normalized such that wc ≥ 0, ∀c = 1, . . . , q and

∑q c=1 wc = 1. Additional restrictions may be

specified by the DM whenever he/she wants to prevent combinations of values for these coefficients that are not aligned with the organization’s goals.

123

An application of value-based DEA to identify… 747

The value-based DEA model avoids the scale dependence problem of the additive DEA model (Gouveia et al. 2008), since all the input and output measures are trans- lated into utility units. Moreover, the weights used in the aggregation gain a specific meaning: they are the scale coefficients of the utility functions. The weights are vari- ables to be set by each DMU to become the best DMU (if possible) or else to minimize the utility difference to the best DMU, rather than being fixed beforehand as in Ali et al. (1995) or Thrall (2000).

The classical decision theory offers some rules, such as min–max and min–max regret, which can be applied to situations where utility functions themselves are uncer- tain. The value-based DEA method uses the min–max regret rule to capture the utility loss associated with choosing an alternative instead of another. The idea is to deter- mine for each alternative the potential regret associated with its choice, noting how much better could another alternative be. For a given weighting vector, the utility loss of choosing one alternative ak instead of a different alternative aj is given by R(ak, aj ) = max j �=k U(aj ) − U(ak).

The maximum regret associated with a given alternative ak, denoted by Rmax(ak), is the maximum loss of opportunity associated with choosing that alternative in an environment where w is interpreted as varying over some suitably specified feasible set: Rmax(ak) = maxw{(max j �=k U(aj )) − U(ak)}.

The min–max regret optimal decision minimizes the worst-case loss with respect to possible realizations of the utility function, that is, alternatives with the minimum Rmax are preferred.

Considering that the alternatives are the DMUs to be evaluated according to q (with q = m + p) criteria, q utility functions must be defined such that the worst level has value 0 and the best level has value 1. Utility functions are monotonic: non-decreasing for outputs and non-increasing for inputs. Hence, after being con- verted into utility values, all factors are treated as outputs to be maximized. For each alternative (DMU), according to the additive MAUT model, the utility obtained is U(DMU j ) =

∑q c=1 wcuc(DMU j ), where wc≥0, ∀c = 1, . . . , q and

∑q c=1 wc = 1.

The scale coefficients w1, . . . , wq are the weights of the utility functions and reflect the utility trade-offs of the DM.

The value-based DEA method (Gouveia et al. 2008) can be extended to consider the super-efficiency concept introduced in DEA by Andersen and Petersen (1993) to further discriminate efficient units (Gouveia et al. 2013). For that purpose, problem (1) is solved (Phase 1):

min dk ,w

dk

s.t. q∑

c=1 wcuc(DMU j ) −

q∑

c=1 wcuc(DMUk) ≤ dk, j = 1, . . . , n; j �= k

q∑

c=1 wc = 1,

wc ≥ 0, ∀c = 1, . . . , q.

(1)

123

748 M. C. Gouveia et al.

In this model, the DMU being evaluated (DMU k) is allowed to choose the weights vector w = (w1, . . . , wq ) that minimizes the score d∗k . This score has a “min–max regret” (utility loss) intuitive meaning (the less, the better). It corresponds to the distance defined by the utility difference to the best of all alternatives (note that the best alternative will also depend on w), excluding itself from the reference set. If the optimal value d∗k of the objective function in (1) is not positive, then the DMU k under evaluation is efficient (there is a weights vector for which it has the best utility, possibly with ties, among all DMUs); otherwise it is inefficient (no matter which weights vector is chosen, there is always a DMU with better utility). Using d∗k , we can assess the extent to which an efficient DMU may worsen its utility while remaining efficient. In case the DMU is inefficient, Phase 2 can be used to find an efficient target.

The details of this process are as follows:

Phase 1 Convert inputs and outputs into utility scales. Compute the efficiency mea- sure, d∗k , for each DMU, k = 1, . . . , n, and the corresponding weighting vector. Phase 2 If d∗k ≥ 0 then solve the “weighted additive” problem (2), using the opti- mal weighting vector resulting from Phase 1, w∗k , and determine the corresponding projected point of the DMU under evaluation.

min λ,s

zk = − q∑

c=1 w

∗ c sc

s.t. n∑

j=1, j �=k λ juc(DMU j ) − sc = uc(DMUk), c = 1, . . . , q

n∑

j=1 λ j = 1,

λ j , sc ≥ 0, j = 1, . . . , k − 1, k + 1, . . . , n; c = 1, . . . , q.

(2)

The variables λ j , j = 1, . . . , k −1, k +1, . . . , n define a convex combination of utility score vectors associated with the n−1 DMUs. The set of efficient DMUs (possibly a single one) defining the convex combination with λ j > 0 are called the “peers” of DMU k under evaluation. The convex combination corresponds to a point on the efficient frontier that is better than DMU k by a difference of value of sc on each criterion c.

3 The Portuguese health care service: primary health care

PHC units are main form of contact of the population with the health system. In 1979, the NHS has been created to provide national, universal, comprehensive and free access to health care, which should be guaranteed by the State. The NHS comprises five health administrations, distributed by the different regions of Portugal (Regional Health Authorities); they develop strategic guidelines; coordinate all aspects of health care provision; supervise management of hospitals and PHC; establish agreements

123

An application of value-based DEA to identify… 749

and protocols with private bodies; and develop a long-term care network (Barros et al. 2011). In the last 40 years, the PHC has undergone successive reform attempts. The 1971–2004 period was problematical, with the passage of laws adopted in successive governments that limited the management autonomy of these structures. In 2005, a new phase in the PHC sector began with the creation of a taskforce to lead the voluntary and independent creation of Family Health Units, with annual individual and group work plans and performance targets (Lapão 2008).

The constitution of the ACES establishes its organization into functional units, namely in Family Health Units, Personalized Health Care Units (both join general practitioners, nurses and administrative staff), community care units, public health units and units of shared care resources. The ACES also provides for the existence of support units, such as the CEO (a general practitioner), the Community Council, the Clinical Council (includes a general practitioner as president, a public health physi- cian, a nurse, and another health professional), the Management Support Unit and the Citizen Office. Although Family Health Units and Personalized Health Care Units are similar in terms of size, they are different because the first are regulated by specific leg- islation that includes rewards associated with achieving the goals contracted, whereas the second are bound by rules approved by the Clinical Council. Note that the admin- istrative autonomy of ACES is closely related to the existence of a Contract-Program to be established between the ACES and the Regional Health Authorities.

Starting in 2010, CEOs/Clinical Councils with the support of Contract Departments of Regional Health Authorities triggered the internal contracting with Family Health Units, which had begun earlier, and the same model was adapted to Personalized Health Care Units. Two planning instruments (the Performance Plan and the Contract- Program) were defined and implemented on pilot basis describing indicators and the internal contracting goals for all the Family Health Units and Personalized Health Care Units, as well as external contracting with ACES. The objectives and goals expressed in the Contract-Program do not necessarily correspond to all performance indicators presented in the Performance Plan. The contracts between the ACES and the Family Health Units and between ACES and Personalized Health Care Units (internal contracting) have in common 15 of the indicators from the Performance Plan. These common indicators are distributed as: four access indicators; eight health care performance indicators; one indicator of perceived quality and two indicators of economic performance. In 2011, reference values were defined for each indicator. These must be updated annually in agreement with the contracting departments and Regional Health Authorities (PHP 2010).

The contracting has instilled a greater sense of demand and accountability of providers and therefore has clear implications for health administration, that is, it has to be able to identify health needs, understand the differences between units, eval- uate, hold responsible and differentiate good practices. Each ACES has a mandate for monitoring and improvement of efficiency of PHC services to make the best possible use of resources. They have only partial autonomy, since financial resources are under the control of the Regional Health Authorities, which are still responsible for plan- ning and resource allocation. However, the ACES has the responsibility to develop their own organizational structure and management tools that better serve to define the clinical and technical governance systems (Lapão and Dussault 2012).

123

750 M. C. Gouveia et al.

4 Data and models developed including the DM’s preferences

This section describes the performance assessment of 12 health units monitored by ACES Baixo Mondego I, in which there was interaction with a GDM: the CEO of the ACES, a member of the Clinical Council and the coordinator of the Management Support Unit. The units to be evaluated are Family Health Units and Personalized Health Care Units, since all of them are in the Contract-Program established between the ACES Baixo Mondego I and Centro Regional Health Authorities. The performance assessment of units is based on the Performance Plan, without distinguishing between Family Health Units and Personalized Health Care Units. Therefore, according to the GDM, the units under evaluation are considered an homogeneous set of DMUs.

The monitoring and follow-up of health units was based on a set of 15 indicators. The units could not select the indicators, but the goals to be achieved on these indicators were negotiated between the health unit and the CEO/Clinical Council, given the health objectives set by Regional Health Authorities and ACES. The data analysed are from 2010 and were extracted from various information systems in the medical office and in the accounting and management department.

The GDM needed a method with the possibility of using multiple factors in an inte- grated model, with an easy and meaningful process to express managerial preferences and the prospect of identifying best practices, sources of inefficiency, gaps relatively to the best practices, and opportunities for improvement. The value-based DEA method was proposed to the GDM, which represents the ACES Baixo Mondego I, to assess the performance of the 12 DMUs (Family Health Units and Personalized Health Care Units) as providing a solution to these requirements. The GDM made clear in the first meeting that it did not intend to obtain a measure of overall performance. The purpose was to construct different models in accordance with the diverse causes of concerns, leading to a deeper understanding of the reasons of inefficiency of some units, and what corrective actions have to be undertaken.

The ACES Baixo Mondego I anticipated that value-based DEA would offer decision support that could contribute to design interventions adapted to the limitations of resources available and the specificities of the community that they serve, including managerial expertise into the analysis.

4.1 Definition of models

After the objectives of this study have been established, we began to define the inputs and outputs for each model taking into account the indicators of the main areas of the Performance Plan. We constructed four different models given the various possible insights resulting from the different perspectives of evaluation used. For brevity, we present the four models but we only describe and discuss the results for models 1 and 2 in detail (see Gouveia 2012 for detailed results for the other models).

The GDM chose four factors to build model 1: number of doctors (xDOC), number of nurses (xNUR), and number of administrative, technical and other support professionals (xADM) as inputs, and number of medical consultations for registered patients (yCONS) as an output. These factors stem from indicators defined in the Performance Plan to

123

An application of value-based DEA to identify… 751

evaluate access: percentage of medical consultations to the user, overall utilization rate of consultations, medical home visits rate per 1000 subscribers, and nurse home visits rate per 1000 registered. The difference between the first and second indicator defined in the Performance Plan (PHP 2010) is that the numerator of the first one is the full-face consultation by the own doctor and the numerator of the second one is the number of registered users with at least one medical consultation; the denominator of both indicators is the number of users registered in the health unit in the period. The GDM agreed that model 1 would address human resources needed for the attendance of registered patients in each unit, i.e. the access to medical consultation for the registered patients in 2010.

Since model 1 only takes into account the number of medical and non-medical staff, the CEO considered it would be interesting to build a new model where the inputs would be costs: total cost billed to the NHS with complementary means of diagnosis and treatment (xCMDT); total medicine costs billed to the NHS (xMED); total cost of human resources (xHR) and medicine costs not billed to the NHS, clinical consumables and other costs (xOC) and the only output would be yCONS. The CEO wanted to know theimpactontheresultsifinadditiontototalcostofhumanresources(includedin xHR) other costs associated with the unit were also considered. This is named henceforward model 2. It is noteworthy that xCMDT and xMED are the two indicators of economic performance of the Contract-Program. To these, the GDM added xHR and xOC.

To complete the study for the ACES Baixo Mondego I, two other models were built to examine perspectives other than access. In a third model, the inputs reflect the costs of providing PHC (the same of model 2), but the outputs are: number of family planning medical consultations; number of maternity health consultations; number of child (aged 0–18) health consultations; number of adult (aged 19–64) health consultation; number of elderly (aged more 65) health consultations and satisfaction with the quality of service. A fourth model links the four areas mentioned in the Performance Plan resulting in nine factors. The inputs are economic indicators of the Performance Plan (the same of model 2) and the outputs are: number of home visits by the doctor; number of doctor consultations in the health unit; number of home visits by the nurse; number of nurse consultations in the health unit and satisfaction with the quality of service. The latter model combines the multiple axes of evaluation, and also reflects the entire medical, nursing and administrative work (for details, see Gouveia 2012).

These inputs and outputs have been widely discussed with the GDM, in accordance with the indicators established in the Performance Plan and are often considered in this type of DEA-based studies (Amado and Santos 2009).

4.2 Elicitation of utility functions

There are several techniques to elicit preference information from a DM/GDM to build utility functions consistent with their answers (see von Winterfeldt and Edwards 1986; Goodwin and Wright 1998). In the value-based DEA method, the purpose of factor conversion into a utility scale (nonlinear utility functions) is to reflect the preferences of the DM/GDM, taking into account the generalization of the DEA methodology pre- sented by Cook and Zhu (2009) that incorporates piecewise linear functions of factors.

123

752 M. C. Gouveia et al.

AsreferredtoinSect.2,theutilityfunctionsaredefinedforeachcriterionbyconverting its range of original performances into the range of utilities [0, 1]. Utility levels 0 and 1 are anchor points defined by the worst and best performances on each criterion and the numbers on the scale do not have an absolute meaning. To construct the utility func- tions, we elicit the difference in the DMU’s value that corresponds to improvements in inputs and outputs, rather than the utility of having these inputs available or outputs produced. In this way, we do not speak of absolute utilities, but rather relative utilities.

The elicitation protocol can be based on comparing the value of increasing an output (or decreasing an input) from a to b versus increasing the same output (or decreasing the same input) from a′ to b′, all other performance levels being equal, and asking the DM/GDM to adjust one of these four figures such that the increase of value would be approximately equal. This is always possible assuming the functions are continuous and monotonic. For example, considering xDOC, the question posed to GDM was: “Is it more valuable to decrease the number of doctors from 20 to 10 or from 10 to 4, all the other performances being equal?” The answer was: “it is more valuable to decrease from 20 to 10.” Then, an adjustment has been made and the question was reformulated as: “Is it more valuable to decrease the number of doctors from 20 to 9 or from 9 to 4, all the other performances being equal?” The answer was: “the value is the same”. This means that uDOC(4) − uDOC(9) = uDOC(9) − uDOC(20), i.e. uDOC(9) = (uDOC(4) + uDOC(20))/2. The same procedure was used to dichotomize the intervals of value [4, 9] and [9, 20].

The GDM was available to answer questions about the differences of value between the performance levels in each factor. We used a piecewise linear approximation to represent the utility functions for most of the factors, and we used other functions (such as logarithmic functions) when the GDM’s answers could be adjusted to predefined curves.

The elicited ranges were chosen to include the observed performance ranges plus or minus the highest tolerance value considered (in this case δ = 20 % to account for uncertainty).

Table 1 summarizes the performance levels corresponding to value 0.25, 0.5, and 0.75 (resulting from this dialogue), such that an improvement from level 0 to level 0.25 corresponds to the same value as an improvement from level 0.25 to 0.5, etc. In Fig. 1, the utility functions for the factors xCMDT, xMED, xHR and xOC are displayed. For example, the medicine costs not billed to the NHS, clinical consumables and other costs (xOC) is a piecewise linear function where: uOC(600,000) − uOC(400,000) = uOC(400,000)−uOC(250,000) = uOC(250,000)−uOC(120,000) = uOC(120,000)− uOC(50,000), all other performance levels being equal. For xCMDT, xMED and xHR, the utility functions were obtained by fitting a logarithmic function to match as well as possible the answers of the GDM. The other inputs and the output utility functions of model 1 (xDOC, xNUR,xADM and yCONS) are also piecewise linear utility functions.

4.3 Elicitation of weight restrictions

In the value-based DEA method, the DMU under evaluation is free to choose the scale coefficients (weights) of the marginal utility functions, so as to become the best DMU

123

An application of value-based DEA to identify… 753

Table 1 Summary of the performance corresponding to different (and equally spaced) utility levels for the factors in model 1

Utility Inputs output

xDOC xNUR xADM yCONS

0 20 20 15 4000

0.25 12 12 10 12,000

0.5 9 8 7 18,000

0.75 6 5 5 22,000

1 4 4 3 25,000

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 200 400 600 800

ut ili

ty

medicine costs not billed to NHS, clinical consumables and other costs (xOC)

Thousands

y = -0.422ln(x) + 6.035 R² = 1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0E+00 5.0E+02 1.0E+03 1.5E+03 2.0E+03

ut ili

ty

total cost billed to the NHS of complementary means of diagnosis and

treatment (euros) (xCMDT)

Thousands

y = -0.519ln(x) + 8.0504 R² = 1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0.0E+00 2.0E+03 4.0E+03 6.0E+03

ut ili

ty

total medicine costs billed to the NHS (euros) (xMED)

Thousands

y = -0.757ln(x) + 11.284 R² = 1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 1000 2000 3000 4000

ut ili

ty

total cost of human resources (euros) (xHR) Thousands

Fig. 1 The utility functions of costs elicited

(if possible) or else to minimize the utility difference to the best DMU, that is, to obtain the best possible efficiency score. In additive MAUT models, all utility functions are added (adjusted by scaling coefficients) regardless of their nature (costs or benefits, monetary or non-monetary). These models add up utility value units (the translation of inputs or outputs into utility values); they do not add up the input and output scales (Keeney and Raiffa 1976; von Winterfeldt and Edwards 1986). However, some factors

123

754 M. C. Gouveia et al.

may be disregarded from the assessment because DMUs can assign zero weights to some factors, in a way incompatible with the GDM’s preferences. So, it is necessary to consider weight restrictions, once these reflect better the organizational objectives and therefore ensure meaningful results closer to what the GDM considers to be the best practices. Several approaches exist to define appropriate weight restrictions, which can be a challenging task (Salo and Hämäläinen 2001; Podinovski 2004).

In the value-based DEA method (Gouveia et al. 2008), the scale coefficients (weights) reflect possible trade-offs of utility between different factors (see Keeney and Raiffa 1976). The assignment of values to the scale coefficients requires a series of judgments by the DM. Direct rating techniques should be avoided, because the value of these coefficients does not reflect the intuitive notion that the DM has of the importance of each criterion. Rather, they are strongly dependent on the performances chosen to represent the levels 0 and 1 on the utility scale. In MCDA, there are several valid protocols to elicit the weight restrictions derived from DM’s preferences (von Winterfeldt and Edwards 1986; Goodwin and Wright 1998). In this case, the swings technique is simple and clear to the DM.

The swing method begins by constructing two extreme hypotheses, P0 and P1, the first one displaying the worst performance (having utility 0) in all criteria scales and the second one the corresponding best performance (having utility 1). The preference elicitation protocol consists in querying the DM to look at the potential gains from moving from P0 to P1 on each criterion and then to decide which one of the criteria he/she prefers to shift to hypothesis P1. Suppose that the transition from P0 to P1 on that criterion is worth 100 units on a hypothetical scale. Then, the DM is asked to give a value (<100) to the second criterion moved to P1, then to the third criterion and so on, until the last criterion is moved to P1.

The procedure used in this work was first to obtain a ranking of weights and then to establish a limit to the ratio between the weights ranked first and last to avoid null weights. Let Wdenote the set of weight vectors compatible with the elicited ranking and limit.

To include the weight restrictions, Phase 1 (see Sect. 2) has the same formulation as (3) but adding the restriction (w1, . . . , wq ) ∈ W . This change in Phase 1 implies changing the formulation of the problem to be solved in Phase 2. It is now necessary to allow slacks to have negative values; otherwise, it might not be possible to keep the optimal utility difference d∗k derived from formulation (3) including the weight restrictions. Note that if slacks are imposed to be positive with the introduction of weight restrictions, then the projection of inefficient units has to be made out of the production possibility set.

5 Results

As previously presented, the GDM proposed two models (1 and 2) to evaluate efficiency in terms of access. Tables 2 and 3 indicate the original data and the corre- sponding utilities.

Table 4 gives the results for model 1 using formulation (3) which allows to rank the DMUs in terms of optimal utility loss d*: DMU7 DMU5 DMU4 DMU3

123

An application of value-based DEA to identify… 755

Table 2 Performances of DMUs in original scales and in utility scales for model 1

DMU Factors in original scales Factors in utility scales

xDOC xNUR xADM yCONS uDOC uNUR uADM uCONS

1 8 8 6 8962 0.583 0.500 0.625 0.155

2 9 9 7 8439 0.500 0.438 0.500 0.139

3 7 5 6 6247 0.667 0.750 0.625 0.070

4 5 6 4 7291 0.875 0.667 0.875 0.103

5 4 5 4.5 5631 1.000 0.750 0.813 0.03

6 10 9 8 11,942 0.417 0.438 0.417 0.248

7 15 14 12 19,356 0.156 0.188 0.150 0.585

8 10 12 10 14,011 0.417 0.250 0.250 0.334

9 6 6 5 7230 0.750 0.667 0.750 0.101

10 9.5 10 8 11,436 0.458 0.375 0.417 0.232

11 13 13.5 10 16,351 0.219 0.203 0.250 0.431

12 14 13 11 14,300 0.188 0.219 0.200 0.346

DMU9 DMU11 DMU8 DMU6 DMU1 DMU10 DMU2 DMU12. The less the value of d* the better, and if d* is negative then the DMU under analysis is efficient (DMUs 7, 5, 4 and 3). The remaining DMUs have d∗ > 0 and hence are not efficient.

The GDM did not agree with null weights in the results, since this corresponds to disregarding some of the factors used in the evaluation. That led to the introduction of weight restrictions. For example, DMU 5 is efficient but all weights were null except w∗DOC. This DMU has only four doctors and is the best on this criterion; therefore, the vector of weights that favoured it most was (1, 0, 0, 0).

In Table 5, the results from Phase 1 for model 2 are presented. For model 2, the ranking of units is (the first seven DMUs are efficient): DMU 9 DMU 7 DMU 5 DMU 4 DMU 1 DMU 2 DMU 3 DMU 8 DMU 10 DMU 11 DMU 12 DMU 6.

The GDM was dissatisfied with the continued existence of units that did not con- sider all factors in their evaluation. In fact, DMU 5 and DMU 7 considered only one of the five factors to be ranked as efficient. Nevertheless, the rank of the units is com- pletely different from model 1, including a higher number of efficient units. It was also explained that the introduction of weights restrictions, as a way of ensuring that all factors are counted in the analysis, would degrade the efficiency of some units, which could move from an efficient status to an inefficient status (Dyson et al. 2001).

According to the procedure explained in Sect. 4.3, weight restrictions were elicited by asking the GDM to compare the “swings” from utility 0 to 1 as depicted in Table 6, for model 1.

The GDM was asked to consider a unit with the performance level 0 for all factors and the question was: “If you could improve one and only one factor to the level 1, which one would it be?” The GDM answer was unanimous: yCONS. This allows

123

756 M. C. Gouveia et al.

T ab

le 3

P er fo rm

an ce s of

D M U s in

or ig in al sc al es

an d in

ut il it y sc al es

fo r m od el 2

D M U

F ac to rs in

or ig in al sc al es

F ac to rs in

ut il it y sc al es

x C M D T

x M E D

x H R

x O C

y C O N S

u C M D T

u M E D

u H R

u O C

u C O N S

1 36

2 ,9 00

1, 11

4 ,9 35

1, 39

8, 03

3 12

3, 38

2 89

62 0. 63

3 0. 82

4 0. 57

2 0. 74

3 0. 15

5

2 46

3, 54

9 1, 19

2 ,3 30

1, 05

0 ,5 58

12 8, 60

6 84

39 0. 52

9 0. 78

9 0. 78

8 0. 73

3 0. 13

9

3 46

6, 56

1 1, 11

3, 73

4 1, 58

9, 81

8 10

1, 75

0 62

47 0. 52

7 0. 82

4 0. 47

5 0. 81

5 0. 07

0

4 27

7, 14

0 1, 32

1, 77

9 1, 00

1, 29

7 15

2 ,9 74

72 91

0. 74

6 0. 73

5 0. 82

5 0. 68

7 0. 10

3

5 23

2 ,5 65

1, 14

7, 29

5 1, 10

1, 20

0 14

1, 86

4 56

31 0. 82

0 0. 80

9 0. 75

3 0. 70

8 0. 05

1

6 61

3, 53

5 2 ,4 91

,9 23

1, 72

4 ,0 28

21 8, 89

0 11

,9 42

0. 41

1 0. 40

6 0. 41

3 0. 56

0 0. 24

8

7 1, 33

2 ,3 17

4 ,3 65

,6 05

2 ,3 68

,4 01

36 0 ,0 11

19 ,3 56

0. 08

4 0. 11

5 0. 17

3 0. 31

7 0. 58

5

8 70

1, 12

1 2 , 2 89

,3 34

2 ,2 25

,3 21

49 1, 14

0 14

,0 11

0. 35

5 0. 45

0 0. 22

0 0. 13

6 0. 33

4

9 31

4 ,0 90

1, 19

6, 35

3 1, 05

4 ,5 64

65 ,4 16

72 30

0. 69

4 0. 78

7 0. 78

5 0. 94

5 0. 10

1

10 60

6, 14

8 2 ,9 54

,0 54

1, 34

8, 65

1 20

3, 09

8 11

,4 36

0. 41

6 0. 31

8 0. 59

9 0. 59

0 0. 23

2

11 1, 00

7, 90

9 3, 35

0 ,3 21

1, 93

3, 79

1 37

1, 46

8 16

,3 51

0. 20

2 0. 25

3 0. 32

6 0. 29

8 0. 43

1

12 70

8, 93

6 2 ,7 93

,6 86

2 ,2 27

,3 92

27 8, 93

6 14

,3 00

0. 35

0 0. 34

7 0. 21

9 0. 45

2 0. 34

6

123

An application of value-based DEA to identify… 757

Table 4 Results from Phase 1 without weight restrictions for model 1

DMU d∗ w∗DOC w ∗ NUR w

∗ ADM w

∗ CONS

1 0.0599 0.000 0.391 0.088 0.521

2 0.1039 0.000 0.391 0.088 0.521

3 −0.0099 0.000 0.487 0.000 0.513 4 −0.0625 0.000 0.000 1.000 0.000 5 −0.1250 1.000 0.000 0.000 0.000 6 0.0529 0.000 0.429 0.053 0.518

7 −0.1535 0.000 0.000 0.000 1.000 8 0.0528 0.387 0.000 0.000 0.613

9 0.0120 0.000 0.391 0.088 0.521

10 0.0869 0.000 0.391 0.088 0.521

11 0.0523 0.000 0.000 0.399 0.601

12 0.1077 0.000 0.429 0.053 0.518

Table 5 Results from Phase 1 without weight restrictions for model 2

DMU d∗ w∗CMDT w ∗ MED w

∗ HR w

∗ OC w

∗ CONS

1 −0.0339 0.112 0.421 0.000 0.000 0.467 2 −0.0226 0.000 0.242 0.186 0.000 0.572 3 −0.0115 0.000 0.846 0.000 0.154 0.000 4 −0.0420 0.205 0.000 0.795 0.000 0.000 5 −0.0740 1.000 0.000 0.000 0.000 0.000 6 0.0504 0.363 0.000 0.000 0.078 0.559

7 −0.1535 0.000 0.000 0.000 0.000 1.000 8 0.0238 0.327 0.096 0.000 0.000 0.577

9 −0.1672 0.000 0.000 0.207 0.793 0.000 10 0.0240 0.052 0.000 0.325 0.064 0.559

11 0.0241 0.029 0.000 0.396 0.000 0.575

12 0.0263 0.380 0.000 0.000 0.055 0.565

Table 6 Extreme performances associated with utility levels 0 and 1 (model 1)

Utility level xDOC xNUR xADM yCONS

u(.) = 0 20 20 15 4000 u(.) = 1 4 4 3 25,000

the inference that wCONS is the highest scaling coefficient. By repeating this ques- tion successively for the remaining factors, we obtained the following ranking of the coefficients of scale (all the answers converged): wCONS ≥ wDOC ≥ wNUR ≥ wADM.

Null weights were avoided by asking an indifference judgment allowing to limit the ratio of the weights ranked first (wCONS) and last (wADM): “What is the highest number h that would allow a DMU with three administrative, technical and other

123

758 M. C. Gouveia et al.

Table 7 Extreme performances associated with utility levels 0 and 1 (model 2)

Utility level xCMDT xMED xHR xOC yCONS

u(.) = 0 1,600,000 5,500,000 3,000,000 600,000 4000 u(.) = 1 150,000 800,000 800,000 50,000 25,000

Table 8 Phase 1 and Phase 2 results under weight restrictions and free slacks (model 1)

DMU Phase 1 Phase 2

d∗ w∗DOC w ∗ NUR w

∗ ADM w

∗ CONS s

∗ DOC s

∗ NUR s

∗ ADM s

∗ CONS

1 0.0715 0.143 0.143 0.143 0.571 0.292 0.167 0.250 −0.052 2 0.1184 0.163 0.163 0.106 0.567 0.375 0.229 0.375 −0.036 3 0.0655 0.170 0.170 0.105 0.556 0.208 −0.083 0.250 0.033 4 −0.0080 0.145 0.145 0.145 0.564 5 −0.0454 0.314 0.314 0.059 0.314 6 0.0788 0.163 0.163 0.106 0.567 0.458 0.229 0.458 −0.145 7 −0.0767 0.120 0.120 0.120 0.639 8 0.0738 0.205 0.109 0.109 0.578 0.458 0.417 0.625 −0.231 9 0.0348 0.163 0.163 0.106 0.567 0.125 −0.000 0.125 0.002 10 0.0912 0.163 0.163 0.106 0.567 0.417 0.292 0.458 −0.130 11 0.0621 0.143 0.143 0.143 0.571 0.656 0.464 0.625 −0.328 12 0.1198 0.163 0.163 0.106 0.567 0.688 0.448 0.675 −0.243

support professionals and 4000 medical consultations for registered patients to be considered as having more value than a unit with 15 administrative, technical and other support professionals and h medical consultations for registered patients?” The answer allows to build the following inequality: wADM uADM(3) + wCONS uCONS(4000) ≥ wADM uADM(15) + wCONS uCONS(h). The elicited bound was h = 10,000. Increasing this amount, the GDM would start hesitating whether the inequality would still hold. Substituting hin the previous expression yields: wCONS ≤ 5.32 wADM.

The same methodology was used for model 2. Table 7 shows the utility levels 0 and 1 for the extreme performance values. Using the same protocols to elicit the weight restrictions as before, the ranking obtained for the scale coefficients was: wMED ≥ wCMDT ≥ wHR ≥ wOC ≥ wCONS. The answer to the question “What is the lowest amount h that would allow a unit with 25,000 medical consultations for registered patients and total medicine costs billed to the NHS of 5.5 million euros to be considered as having more value than a unit with 4000 medical consultations for registered patients and total medicine costs billed to the NHS of h?” was h = 2.5 million euros. This answer is translated as: wCONS uCONS(25,000) + wMED uMED(5,500,000) ≥ wCONS uCONS(4000) + wMED uMED(h). Substituting hyields: wMED ≤ 2.47 wCONS.

The results for Phase 1 and Phase 2 with the addition of weight restrictions and free slacks are presented in Table 8 for model 1. There are no longer null weights and being the best DMU in some of the factors no longer guarantees being considered efficient.

123

An application of value-based DEA to identify… 759

The ranking induced by d∗, for model 1 is: DMU 7 DMU 5 DMU 4 DMU 9 DMU 11 DMU 3 DMU 1 DMU 8 DMU 6 DMU 10 DMU 2 DMU 12. The first three are efficient.

Table 9 shows the results obtained in Phase 1 and Phase 2 considering the weight restrictions and free slacks for model 2. In model 2, the d∗ value obtained led to the following ranking: DMU 9 DMU 5 DMU 4 DMU 2 DMU 1 DMU 3 DMU 10 DMU 6 DMU 12 DMU 8 DMU 11 DMU 7, and only the first two DMUs are efficient. Making the comparison of model 1 results without weight restrictions and with weight restrictions, DMU 3 loses the efficient status but the remaining efficient DMUs maintain the same ranking order in terms of efficiency. The last three inefficient units in both rankings remain the same. Regarding model 2, which had seven efficient units, only DMU 9 and DMU 5 are now efficient, which is better suited to reality according to the GDM’s opinion. The position of the units in the two rankings is very different. In fact, the introduction of weight restrictions in model 2 led to major changes in the evaluation of DMUs.

Considering now model 1 and model 2 with weight restrictions, the GDM attached significance to the fact that DMU 5 remained efficient in both models, since their aim is to assess users’ access, given the resources consumed in each unit. This result is consistent with the perception the GDM had of the operation of this unit in particular. DMU 9 is the first of the efficient units in model 2, but the first inefficient one in model 1, which makes sense because of the importance that the GDM gave to costs (more weight) in model 2 to the detriment of yCONS. Note that DMU 9 has more human resources than DMU 5; however, it has less total cost of human resources and less than half in medicine costs not billed to the NHS, clinical consumables and other costs. A major turnaround happens with DMU 7, which is efficient in model 1 with and without restrictions (the best of all units) and also in model 2 without weight restrictions, but falls to the last position in the ranking in model 2 with weight restrictions. The explanation is that although DMU 7 has more yCONS than any other unit, it spends a lot of money in medicines and xCMDT and these two factors have more weight for the GDM. This was welcomed by the GDM, knowing that the unit in question provides health services to a very elderly population, which in turn consumes more in medicines and xCMDT.

6 Improvements of inefficient units and further developments

Phase 2 uses the optimal weighting vector w∗ ∈ W , to solve the LP problem (2) with free slacks. The solution is a proposal of an efficiency target (projection) for each inefficient DMU. To attain an efficient status, these inefficient DMUs must change their utility in each factor by the amount indicated by s∗, considering the two models. Note that not all the proposed changes correspond to improvements in all the factors since some of the slacks are negative. For an inefficient unit to “emulate” its peers on the efficient frontier, if it has a negative slack for an input factor this corresponds to an increase of the input, and if it has a negative slack for an output factor this corresponds to a decrease of the input. This is an interesting result since for model 1 it can be shown that some units (e.g. DMU 3) have an excessive number of doctors and few nurses

123

760 M. C. Gouveia et al.

T ab

le 9

P ha se

1 an d P ha se

2 re su lt s un de r w ei gh t re st ri ct io ns

an d fr ee

sl ac ks

(m od el 2)

D M U

P ha se

1 P ha se

2

d ∗

w ∗ CM

D T

w ∗ M E D

w ∗ HR

w ∗ OC

w ∗ CO

N S

s∗ C M D T

s∗ M E D

s∗ H R

s∗ O C

s∗ C O N S

1 0 .0 51

2 0. 15

5 0. 38

2 0. 15

5 0. 15

5 0. 15

5 0 .0 61

−0 .0 37

0 .2 13

0. 20

1 −0

.0 54

2 0 .0 51

1 0. 15

5 0. 38

2 0. 15

5 0. 15

5 0. 15

5 0 .1 64

−0 .0 02

−0 .0 03

0. 21

1 −0

.0 38

3 0 .0 84

5 0. 15

5 0. 38

2 0. 15

5 0. 15

5 0. 15

5 0 .1 67

−0 .0 37

0 .3 11

0. 13

0 0 .0 31

4 0 .0 16

7 0. 26

2 0. 26

2 0. 26

2 0. 10

6 0. 10

6 −0

.0 53

0 .0 52

−0 .0 39

0. 25

8 −0

.0 02

5 −0

.0 05

9 0. 31

1 0. 31

1 0. 12

6 0. 12

6 0. 12

6

6 0 .2 54

1 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0 .2 83

0 .3 81

0 .3 72

0. 38

5 −0

.1 47

7 0 .4 08

5 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0 .6 10

0 .6 72

0 .6 12

0. 62

8 −0

.4 84

8 0 .3 54

7 0. 27

7 0. 32

6 0. 13

2 0. 13

2 0. 13

2 0 .3 39

0 .3 37

0 . 5 65

0. 80

9 −0

.2 33

9 −0

.0 34

2 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0. 20

0

10 0 .2 31

2 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0 .2 77

0 .4 69

0 .1 86

0. 35

5 −0

.1 31

11 0 .3 60

5 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0 .4 92

0 .5 34

0 .4 59

0. 64

7 −0

.3 30

12 0 .3 19

6 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0. 20

0 0 .3 44

0 .4 40

0 .5 66

0. 49

3 −0

.2 45

123

An application of value-based DEA to identify… 761

and to be efficient those units should decrease the number of doctors, even if at the same time they increase the number of nurses. As we have anticipated, the GDM did not fully understand the reduction of yCONS proposed for most of the inefficient units. Additionally, the increase in costs and the yCONS reduction proposed in the results of Phase 2 of model 2 were unacceptable to the GDM. It was necessary to propose another solution, but these preliminary results were instrumental in helping the GDM discuss what improvements should be sought for inefficient DMUs.

Let us consider that u∗k is the utility of the best DMU using the optimal weighting vector w∗ ∈ W resulting from (1) with the weight restrictions added, i.e. the utility value that an inefficient DMU k ought to achieve:

u∗k = q∑

c=1 w

∗ k uc(DMUk) + d∗k . (3)

The LP problem solved in Phase 2 of the value-based DEA method admits alter- native optimal targets, each of which corresponds to a different way of closing the value gap d∗k . For this case, the GDM intends to constrain the efficiency targets that are proposed to achieve u∗k, not only to avoid those targets that imply an increase of inputs or a decrease of outputs, but also to choose which factors can be changed, given the characteristics of each unit.

Let S< = {c ∈ {1, . . . , q} : s∗c < 0 in Phase 2} denote the negative slacks in the optimal solution obtained for (2) with free slacks; these slacks will now become null constants. Let S≥ = {c ∈ {1, . . . , q} : s∗c ≥ 0 in Phase 2} denote the remaining slacks, which will be considered as non-negative variables. Therefore, a formulation that yields an alternative target that maintains or improves the performance of all factors can be obtained by solving the LP model (4) in which the maximum slack (in terms of value) shall be minimized to achieve the global utility value target. No negative slacks are allowed, but the target will no longer be a convex combination of the observed DMUs (for details, see Almeida and Dias 2012).

The formulation below differs from the one used by Almeida and Dias (2012) in that itintroducesanewparameter σc (apositivevalue)foreachfactortoboundthevaluethat a slack may have. The aim is to avoid setting unrealistic targets, allowing the proposals to be better tailored to the GDM’s requirements. The targets will never exceed the value 1 in any factor due to the constraint uc(DMUk) + sc ≤ 1, c = 1, . . . , q. This ensures the utility function does not “spill over” outside the performances ranges elicited.

min ρk ,s

ρk

s.t. ∑

c∈S≥ w

∗ c sc = d∗k

uc(DMUk) + sc ≤ 1, c = 1, . . . , q sc − σc ≤ 0, c ∈ S≥ ρk ≥ sc ≥ 0, c = 1, . . . , q.

(4)

For model 1, all the inefficient units had chosen the DMU 4 as peer and the GDM began by analysing the results in terms of proposals to improve the inefficient units,

123

762 M. C. Gouveia et al.

Table 10 Improvement proposals for inefficient units in original scales (model 1)

DMU Proposal 1 Proposal 2 Proposal 3

xDOC xNUR xADM yCONS xDOC xNUR xADM yCONS xDOC xNUR xADM yCONS

1 6 6 4.5 8962 8 8 6 12,708 7 7 5.5 11,248

2 6 5 5 8439 9 9 7 14,340 7.5 7.5 6 12,156

3 6 5 5.5 8752 7 5 6 10,000 6.5 4.5 5.5 8336

6 8 6.5 6 11,942 9 9 8 14,126 9 8 7 13,836

8 8 9 8 14,011 10 11 10 16,328 9 11 9 15,780

9 5.5 6 4.5 8560 6 6 5 9168 6 5.5 4.5 8336

10 7 7 6 11,436 9.5 9 8 15,620 8.5 8.5 7 13,761

11 10.5 10.5 8.5 16,351 12 12 10 18,632 11.5 12 9.5 17,820

12 9.5 8 7.5 14,300 12 12 11 17,830 11.5 10.5 9.0 17,172

blocking changes in the factors that had negative slacks in Phase 2 (see proposal 1 in Table 10). Given the characteristics of certain units, the GDM deemed some of the proposed reductions in the number of medical and non-medical staff were not reasonable. Thus, the proposal emerged to maintain the original performance of the input factors for some units, or to limit the reduction of those factors for other units, in model 1. The gap between the unit under evaluation and DMU 4 is closed by increasing the output factor (see proposal 2 in Table 10). For some units, the increase in the number of medical consultations for registered patients was considered to be achievable. It is well known that not all registered patients attend the health centre regularly, thereby hindering continuity of the relationship established between the patient and the general practitioner [in 2005, approximately 60 % of the registered patients did not use the services of the health centre (IGIF 2006)]. For other units, the proposed numbers of medical consultations for registered patients would be impossible to achieve. That led to formulation (5) in which the value gap relatively to the efficient peer is distributed in a balanced way on all the factors (see proposal 3 in Table 10).

min ρk ,s

ρk

s.t. q∑

c=1 w

∗ c sc = d∗k

uc(DMUk) + sc ≤ 1, c = 1, . . . , q sc − σc ≤ 0, c = 1, . . . , q ρk ≥ sc ≥ 0, c = 1, . . . , q.

(5)

The improvements to all the scenarios referred to above are displayed in Table 10, with the factors that did not change in bold. The GDM was very pleased to be able to choose in each case the improvement proposal that they felt was most appropriate knowing the capabilities and context of each unit.

The proposed improvements to model 2 are presented in Table 11. Taking into account the model at stake and the units being assessed, the GDM was satisfied with

123

An application of value-based DEA to identify… 763

T ab

le 11

Im pr ov em

en t pr op

os al s fo r in ef fi ci en t un

it s in

or ig in al sc al es

(m od

el 2)

D M U s

P ro po sa l 1

P ro po sa l 2

x C M D T

x M E D

x H R

x O C

y C O N S

x C M D T

x M E D

x H R

x O C

y C O N S

1 27

9, 69

0 1,

11 4,

93 5

1, 20

9, 77

0 91

02 0

89 62

32 1, 66

0 1, 01

0 ,8 95

1, 30

7, 83

0 10

7, 54

0 10

,5 76

2 31

3, 39

0 1,

19 2,

33 0

1, 05

0, 55

8 78

,4 20

84 39

41 1, 54

0 1, 08

1, 47

0 98

3, 17

0 11

0 ,3 40

10 ,0 70

3 33

8, 08

0 1,

11 3,

73 4

1, 00

1, 29

7 63

58 0

10 ,6 10

38 2, 41

0 94

6, 84

0 14

23 21

0 78

,1 40

89 75

4 27

7, 14

0 1, 21

1, 67

0 1,

00 1,

29 7

12 9, 62

0 72

91 26

6, 75

0 12

81 30

0 98

0 ,5 00

14 4 ,7 00

78 24

6 28

9, 13

0 1, 34

9, 72

0 1, 13

2 ,4 50

84 ,3 00

11 ,9

42 33

5, 68

0 1, 52

6, 86

0 1, 23

2 ,3 50

10 2 ,2 20

18 ,0 40

7 39

9, 00

0 1, 63

6, 53

0 1, 20

8, 17

0 98

,8 60

19 ,3

56 50

8, 19

0 1, 99

1, 94

0 1, 38

2 ,4 40

13 3, 78

0 24

,8 98

8 26

6, 75

0 1, 04

2, 59

0 1, 29

7, 51

0 22

6, 86

0 14

,0 11

30 3, 16

0 1, 15

6, 91

0 1, 39

3, 44

0 25

5, 70

0 21

,0 00

10 30

6, 05

0 1, 69

4, 29

0 92

1, 55

0 84

,0 20

11 ,4

36 35

1, 05

0 1, 89

4 ,6 10

99 4 ,9 30

10 0 ,2 60

17 ,1 24

11 34

7, 01

0 1, 40

8, 17

0 1, 06

7, 09

0 12

1, 30

0 16

,3 51

42 9, 50

0 1, 67

4 ,8 10

1, 20

1, 81

0 16

8, 10

0 22

,4 98

12 27

5, 09

0 1, 29

6, 20

0 1, 31

4 ,7 60

91 ,8 60

14 ,3

00 33

2, 51

0 1, 50

9, 31

0 1, 46

1, 31

0 11

4 ,2 60

20 ,6 32

123

764 M. C. Gouveia et al.

Proposals 1 and 2. In Proposal 1, changes to achieve the same global utility as the peer are made on the factors that have non-negative slacks in Phase 2. In Proposal 2, a balanced distribution of the gap value relative to the peer is made by changing all factors (inputs and outputs), which the GDM felt more adequate to their needs.

7 Concluding remarks

This study reports an intervention in which analysts and health experts worked together in an assessment of PHC units, seeking to identify best practices and inform decision- making. The participation of the three elements constituting the GDM was important at different times. The two health professionals had a more active intervention in the construction of the assessment models and utility functions, as well as in restrictions setting, while the coordinator of the Management Support Unit had his main interven- tion on building the assessment models and on the collection of data and information. The three members performed the discussion of results. Five meetings were carried out with the three DMs together in most of them. As the roles of each DMs are well defined in the ACES, the interaction process was always held in a cordial atmosphere and there was rarely room for disagreement. The value-based DEA method was used as a modelling basis and then tailored to the requirements of this application, namely to define realistic targets (by the introduction of a new parameter into the model), to allow ranking the DMUs.

The study to assess the performance of PHC units started with the choice by the GDM of factors to include into the models, namely regarding access. Two models to assess PHC units were created as a result of discussions. The incorporation of the GDM’s preferences in those models was made possible by the conversion of input and output factors into utility, which required interpreting the utility functions as devices to compare the change in value for the GDM that corresponds to performance differences. Models became more complete and credible for the GDM only after the introduction of weight restrictions that were elicited using the swings technique to draw a ranking of weights. Moreover, a trade-off question was used to limit the ratio between the weights ranked first and last and consequently avoid null weights. The effort required from the DMs depends on the number of factors. With a high number of factors, it can be more difficult for DMs to provide a ranking for the respective weights. However, typically the number of factors used in DEA applications is not large.

A ranking of all the units and benchmarks was identified for each model and it was possible to conclude that the Family Health Units are not worst or better than Person- alized Health Care Units. In the access evaluation model with and without restrictions on weights, the results showed that there are efficient health units of both types. How- ever, in the Model 1 results, the efficient units are mostly Personalized Health Care Units, whether considering or not restrictions on weights. The improvement propos- als to inefficient health centres emerged taking advantage of the GDM’s feedback on the model results and their expert knowledge about the units, to obtain more realistic targets. For this purpose, a new model was developed.

Our findings helped the GDM in the identification of best practices, sources of inefficiency and gaps to the best practices, as well as possibilities and opportunities

123

An application of value-based DEA to identify… 765

for improvement. The results were useful not only for the introduction of corrective measures, but also to inform decisions about the goals of PIs in the future. It was also possible to conclude that there are units that given their characteristics, such as having more elderly registered users who in turn have more chronic diseases, benefited from the inclusion of indicators that reflect that. In the future, it would be interesting to include indicators chosen by the health units themselves and evaluate in the same group the ones that chose the same indicators (or better, to allow comparisons, DMUs would choose between a small set of indicator groups).

This application to a real-world problem allowed ascertaining the value of the proposed methodology in practice. The GDM easily understood the methodology and the receptivity increased during the interaction process. The most prized results were the complete ranking of the units and the different proposals for improvement, since the GDM had then the possibility to choose how to improve each unit taking into account its specific characteristics. The answers to the trade-off questions were the methodological aspect that was received with more difficulty by the GDM.

In the future, it would be interesting to develop DEA models capable of helping the PHC appraise investments in the production of outcomes.

Acknowledgments This work has been partially supported by the Portuguese Foundation for Science and Technology under project Grant UID/MULTI/00308/2013. Authors are also indebted to the anonymous reviewers and the area editor for their constructive comments and suggestions on earlier versions of this paper.

References

Ali AI, Lerme CS, Seiford L (1995) Components of efficiency evaluation in data envelopment analysis. Eur J Oper Res 80:462–473

Almeida PN, Dias LC (2012) Value-based DEA models: application-driven developments. J Oper Res Soc 63:16–27

Amado CAF, Dyson RG (2008) On comparing the performance of primary care providers. Eur J Oper Res 185:915–932

Amado CAF, Santos SP (2009) Challenges for performance assessment and improvement in primary health care: the case of the Portuguese health centres. Health Policy 91:43–56

Andersen P, Petersen NC (1993) A procedure for ranking efficient units in data envelopment analysis. Manag Sci 39:1261–1264

Athanassopoulos AD, Podinovski VV (1997) Dominance and potential optimality multiple criteria models decision analysis with imprecise information. J Oper Res Soc 48:142–150

Baker R (1992) General practice in Gloucestershire, Avon and Somerset: explaining variations in standards. Br J Gen Pract 42:415–418

Barros PP, Machado SR, Simões JA (2011) Portugal health system review. Health Syst Transit 13:1–156 Campbell SM, Roland MO, Quayle JA, Buetow SA, Shekelle PG (1998) Quality indicators for general

practice: which ones can general practitioners and health authority managers agree are important and how useful are they? J Public Health Med 20:414–421

Carter M, Hans EW, Kolisch R (2012) Health care operations management. OR Spectr 34:315–317 Charnes A, Cooper WW, Rhodes E (1978) Measuring the efficiency of decision making units. Eur J Oper

Res 2:429–444 Cook WD, Zhu J (2009) Piecewise linear output measures in DEA (third revision). Eur J Oper Res 197:312–

319 Cooper WW, Seiford LM, Zhu J (2004) Handbook on data envelopment analysis. Kluwer Academic Pub-

lishers, Boston

123

766 M. C. Gouveia et al.

Dias LC, Clímaco JN (2000) Additive aggregation with variable interdependent parameters: the VIP analysis software. J Oper Res Soc 51:1070–1082

Donabedian A (1980) The definition of quality and approaches to its assessment. Health Administration Press, Ann Arbor

Dyson RG, Allen R, Camanho AS, Podinovski VV, Sarrico CS, Shale EA (2001) Pitfalls and protocols in DEA. Eur J Oper Res 132:245–259

Ferreira C, Marques RC, Nicola P (2013) On evaluating health centers groups in Lisbon and Tagus Valley: efficiency, equity and quality. BMC Health Serv Res 13:529

Giuffrida A (1999) Productivity and efficiency changes in primary care: a Malmquist index approach. Health Care Manag Sci 2:11–26

Giuffrida A, Gravelle H, Sutton M (2000) Efficiency and administrative costs in primary care. J Health Econ 19:983–1006

Golany B (1988) An interactive MOLP procedure for the extension of data envelopment analysis to effec- tiveness analysis. J Oper Res Soc 39:725–734

Goodwin P, Wright G (1998) Decision analysis for management judgment, 2nd edn. Wiley, Chichester GouveiaMC(2012) Efficiency evaluation with aDEA-MCDAmethod with incorporation of preferencesand

data uncertainty. Ph.D. thesis, University of Coimbra. http://hdl.handle.net/10316/21070. Accessed 18 June 2014

Gouveia MC, Dias LC, Antunes CH (2008) Additive DEA based on MCDA with imprecise information. J Oper Res Soc 59:54–63

Gouveia MC, Dias LC, Antunes CH (2013) Super-efficiency and stability intervals in additive DEA. J Oper Res Soc 64:86–96

Gouveia MC, Dias LC, Antunes CH, Boucinha J, Inácio CF (2015) Benchmarking of maintenance and outage repair in an electricity distribution company using the value-based DEA method. Omega 53:104–114

Halme M, Joro T, Korhonen P, Salo SE, Wallenius J (1999) A value efficiency approach to incorporating preference information in data envelopment analysis. Manag Sci 45:103–115

Hollingsworth B, Dawson PJ, Maniadakis N (1999) Efficiency measurement of health care: a review of non-parametric methods and applications. Health Care Manag Sci 2:161–172

Huang YG, McLaughlin CP (1989) Relative efficiency in rural primary health care: an application of data envelopment analysis. Health Serv Res 24:143–158

IGIF (2006) Estatística do Movimento Assistencial dos Centros de Saúde em 2005. Instituto de Gestão Informática e Financeira da Saúde (in Portuguese)

Joro T, Korhonen P, Wallenius J (1998) Structural comparison of data envelopment analysis and multiple objective linear programming. Manag Sci 44:962–970

Keeney RL, Raiffa H (1976) Decisions with multiple objectives: preferences and value tradeoff. Wiley, New York

Kirigia JM, Emrouznejad A, Sambo LG, Munguti N, Liambila W (2004) Using data envelopment analysis to measure the technical efficiency of public health centers in Kenya. J Med Syst 28:155–166

Lapão LV (2008) The role of complexity and innovation in the new primary care governance model in Portugal. Innov J-Spec Issue Complex 13(3):1–12. Article 8

Lapão LV, Dussault G (2012) From policy to reality: clinical managers’ views of the organizational chal- lenges of primary care reform in Portugal. Int J Health Plan Manag 27:295–307

Linna M, Nordblad A, Koivu M (2003) Technical and cost efficiency of oral health care provision in Finnish health centres. Soc Sci Med 56:343–353

Mavrotas G, Trifillis P (2006) Multicriteria decision analysis with minimum information: combining DEA with MAVT. Comput Oper Res 33:2083–2098

McColl A, Roderick P, Gabbay J, Smith H, Moore M (1998) Performance indicators for primary care groups: an evidence based approach. Br Med J 317:1354–1360

Milliken O, Devlin RA, Barham V, Hogg W, Dahrouge S, Russell G (2011) Comparative efficiency assess- ment of primary care service delivery models using data envelopment analysis. Can Public Policy 37:85–109

Nuti S, Daraio C, Speroni C, Vainieri M (2011) Relationships between technical efficiency and the quality and costs of health care in Italy. Int J Qual Health Care 23:324–330

PHP (2010) Metodologia de Contratualização. Administração Central do Sistema de Saúde, Coordenação Estratégica dos CSP (in Portuguese)

123

An application of value-based DEA to identify… 767

Podinovski VV (2004) Production trade-offs and weight restrictions in data envelopment analysis. J Oper Res Soc 55:1311–1322

Proctor S, Campbell J (1999) A developmental performance framework for primary care. Int J Health Qual Assur 12:279–286

Rosenman R, Friesner D (2004) Scope and scale inefficiencies in physician practices. Health Econ 13:1091– 1116

Rouse P, Harrison J, Turner N (2011) Cost and performance: complements for improvement. J Med Syst 35:1063–1074

Salinas-Jimenez J, Smith P (1996) Data envelopment analysis applied to quality in primary health care. Ann Oper Res 67:141–161

Salo AA, Hämäläinen RP (2001) Preference ratios in multiattribute evaluation (PRIME)—elicitation and decision procedures under incomplete information. IEEE Trans Syst Man Cybern Part A 31:533–545

Sherman HD, Zhu J (2006) Improving service performance using data envelopment analysis. Springer, New York

Sommersguter-Reichmann M (2003) Analysing hospital productivity changes using non-parametric approaches. OR Spectr 25:145–160

Thanassoulis E, Boussofiane A, Dyson RG (1996) A comparison of data envelopment analysis and ratio analysis as tools for performance assessment. Omega 24:229–244

Thanassoulis E, Portela MC, Allen R (2004) Incorporating value judgments in DEA. In: Cooper WW, Seiford LM, Zhu J (eds) Handbook on data envelopment analysis. Kluwer, Boston, pp 99–137

Thanassoulis E, Portela MC, Graveney M (2012) Estimating the scope for savings in referrals and drug prescription costs in the General Practice units of a UK Primary Care Trust. Eur J Oper Res 221:432– 444

Thrall RM (2000) Measures in DEA with an application to the Malmquist index. J Product Anal 13:125–137 van den Hombergh P, Grol R, van den Hoogen HJM, van den Bosch WJHM (1998) Assessment of man-

agement in general practice: validation of a practice visit method. Br J Gen Pract 48:1743–1750 von Winterfeldt D, Edwards W (1986) Decision analysis behavioral research. Cambridge University Press,

New York Wagner J, Shimshak D, Novak M (2003) Advances in physician profiling: the use of DEA. Socio-Econ Plan

Sci 37:141–163

123

OR Spectrum is a copyright of Springer, 2016. All Rights Reserved.