Quality Improvement

International Journal of Public Administration

ISSN: 0190-0692 (Print) 1532-4265 (Online) Journal homepage: https://www.tandfonline.com/loi/lpad20

Management Tools for Quality Performance Improvement in Italian Hospitals

Simone Fanelli, Gianluca Lanza & Antonello Zangrandi

To cite this article: Simone Fanelli, Gianluca Lanza & Antonello Zangrandi (2017) Management Tools for Quality Performance Improvement in Italian Hospitals, International Journal of Public Administration, 40:10, 808-819, DOI: 10.1080/01900692.2017.1280821

To link to this article: https://doi.org/10.1080/01900692.2017.1280821

Published online: 07 Feb 2017.

Submit your article to this journal

Article views: 350

View related articles

View Crossmark data

Citing articles: 6 View citing articles

Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=lpad20

INTERNATIONAL JOURNAL OF PUBLIC ADMINISTRATION 2017, VOL. 40, NO. 10, 808–819 https://doi.org/10.1080/01900692.2017.1280821

Management Tools for Quality Performance Improvement in Italian Hospitals Simone Fanelli, Gianluca Lanza, and Antonello Zangrandi

Department of Public Management, University of Parma, Italy

ABSTRACT Leadership plays a decisive role in improving the quality performance in healthcare organizations. Research examines how Italian hospitals generate conditions to support leadership, at both top management and department/unit level, in improving quality. It focuses on the role of the management in processes for the delegation of responsibilities, and tools implemented to facilitate clinical leadership. Data show that: mandate is linked to full accountability; top manage- ment supports leadership for reinforcing outputs; and evaluation processes are systematically implemented. Furthermore, data show that leaders require performance systems to enhance clinical professionalism and to commit to improving performance.

Keywords Hospitals; leadership; management tools; quality performance

Introduction

Hospitals all over the world face strong pressure from a multiplicity of stakeholders to improve their perfor- mance. However, hospital performance is not a simple and one-dimensional concept; numerous elements must be taken in account. First, different stakeholder groups perceive it differently. Tangsoc and Bautista (2016) identify three such groups, each having a differ- ent perspective on the main performance dimensions. The first is patients, the main service recipients, who seek service satisfaction through effective and quality healthcare services, leading to better health outcomes (Martinez & Martinez, 2010). Patients emphasize cour- tesy, communication, and responsiveness. The second stakeholder group is healthcare professionals, who are the direct service providers. They emphasize the tech- nical attributes including staff competences and patient outcomes. The third group is hospital management, who are the indirect service providers. Management focus on transversal competences, understanding cus- tomers, and collaboration.

The second aspect of hospital performance is that it concerns different areas, including clinical quality, enhancement of services, expansion of access, and cost reduction (Curtright, Stolp-Smith, & Edell, 2000). In general, in industrialized countries, there is demand for more and higher standard health services, along with the need to monitor and limit costs, so that man- agement is obliged to pay increasing attention to the ratio between resources used and healthcare quality (Langlois et al., 2015). Performance evaluation has

thus become a key aspect of running a healthcare organization. Feedback on processes and results is the key to identifying critical areas and implementing improvement procedures (Morse, Koven, Mundt, & Gohmann, 2008; Torok et al., 2016; van Schoten, de Blok, Spreeuwenberg, Groenewegen, & Wagner, 2016). The literature widely recognizes the value of using performance indicators to measure performance and create benchmarks for performance improvement (Berwick, James, & Coye, 2003; de Korne et al., 2012; van Veen-Berkx et al., 2016).

In fact, over time there has been a proliferation of management tools for improving performance, moni- toring and evaluating results, and reporting them to various stakeholder groups. These tools have become increasingly sophisticated and precise to take account of the multidimensional nature of performance and the different types of requirement for information.

Many researchers have thus identified and developed complex approaches for evaluating the performance of healthcare organizations, and these are now outlined.

Quinn and Rohrbaugh (1981) define a Competing Values Framework of four quadrants formed by the intersection of two dimensions. The horizontal dimen- sion represents the internal/external dimension of the organization and the vertical dimension represents the dimension of control/flexibility of the organization. Each quadrant thus measures a dominant conceptual model of organizational performance: rational goal (goal achievement), human relations (participation and involvement of staff), open systems (adaptation to

CONTACT Simone Fanelli simone.fanelli@unipr.it Department of Public Management, University of Parma, Via J.F. Kennedy n. 6, Parma, 43125, Italy. © 2017 Taylor & Francis Group, LLC

809 INTERNATIONAL JOURNAL OF PUBLIC ADMINISTRATION

economic, social, and political environments), and internal process (reliability, stability, and continuity). Keegan, Eiler, and Jones (1989) present a balanced performance measurement matrix, and distinguish per- formance indicators into financial versus nonfinancial and internal versus external. An important aspect of the model is the use of the key metric approach and the “Determine and Decompose” method. This involves decomposing departments into functional equivalents and assessing how the departments support the busi- ness. Azzone, Masella, and Bertelè (1991) propose a detailed and specific performance measurement frame- work based on time. These measures consider internal configuration and external configuration as dimensions of performance that reflect the efficiency and effective- ness of the organization. This framework has the advantage of taking into account different ways the organization can make use of time to gain competitive advantage and the main activities critical for this out- come. Lynch and Cross (1991) define the performance pyramid system, an approach that links the strategic view of performance with operational processes. It shows how the top management informs the organiza- tion of targets and the performance measures to be used at each level, and monitors performance at differ- ent levels of the organization. Kaplan and Norton (1992) present a balanced scorecard framework for measuring the performance of an organization, whereby management monitors performance in four key dimensions: financial perspective, internal perspec- tive, customer perspective, and learning and growth perspective. Brown (1996) splits the process of perfor- mance creation into four steps: input, processes, out- put, and outcome. Each stage is the driver of performance for the next, and the approach develops the concept of linking measures through cause and effect relationships. Finally, Neely, Adams, and Crowe (2001) describe the performance prism. This consists of five integrated parts that identify areas for organiza- tions to address: stakeholder satisfaction, strategies, processes, capabilities, and stakeholder contribution. The critical and unique aspect of the performance prism is the reorganization of the reciprocal relation- ship between the stakeholder and the organization.

The various approaches outlined above have in com- mon that they take into account the multidimensional nature of performance. For hospitals, dimensions of per- formance are clinical effectiveness, safety, patient cente- redness, responsive governance, staff orientation, and efficiency (Veillard et al., 2005). These dimensions are of course considered interdependent and need to be assessed simultaneously, so the main domains of perfor- mance measurement are quality of care, cost and

efficiency, and patient satisfaction (Nerenz & Neil, 2001). All these domains are important, but in healthcare the quality of care is particularly so (Burstin, Leatherman, & Goldmann, 2016; Nwabueze & Mileski, 2008). This domain focuses on the clinical content of care provided for defined groups of patients. It can however be con- sidered as a heterogeneous measurement domain because there are countless quality-of-care measures available. For this reason, structures have to be imposed in order to design a rational system. Donabedian (1988, p. 1745) subdivides quality measures and concepts into “struc- ture,” “process,” and “outcome.” This approach has formed the conceptual foundation for quality measure- ment in this field. Outcome, or the effect on the health of the patient, is most heavily emphasized in health organi- zations, but tends to be difficult to measure and is also difficult to link directly to specific intervention carried out by the hospital. So process measures are typically selected because scientific research has shown a particu- lar practice to be associated with favorable patient out- comes. Joint Commission International (JCI), for example, uses process measures as proxies of outcome to assess quality of care in hospitals; the quality-of-care processes in fact determine the final outcomes (Beattie & Mackway-Jones, 2004). Finally, structure is a key element in care quality: hospitals need to have good facilities, qualified staff in appropriate ratio, and a sound organiza- tional structure that encourages good governance and performance. Most measures in the Joint Commission accreditation system and in the National Committee on Quality Assurance (NCQA) accreditation system are structural measures.

This article focuses on the quality of care as a key dimension in hospital performance and particularly on structure and processes as key components in service- quality improvement. The research was conducted in hospitals following performance improvement strategies toward the high standards of care quality and which are members of an international network. The aim was to describe how these hospitals use assessment tools to support quality improvement at both top management and department/unit level. It examines to what extent the top management and department/unit chiefs evaluate the management tools able to influence professional beha- viors. The results of the study provide useful indications for policy-makers and management aiming to develop systems for the improvement of quality performance.

The impact of leadership on quality performance

In hospitals, personnel plays a key role at the level of both structure and processes in generating qualitative

810 S. FANELLI ET AL.

services and better outcomes. A hospital with a valid structure relies on a group of professionals with clearly delegated roles and responsibilities. Leadership style is particularly important in creating favorable conditions and leading personnel toward the desired outcome (Khaliq, Walston, & Thompson, 2007). Among organi- zations, in fact, healthcare organizations are the most complex to run (Drucker, 2012) and increasing pro- blems facing hospitals have led to the development of the management culture (Griffith, 2007). Leadership, or “the ability to inspire individual and organizational excellence, to create and attain a shared vision, and to successfully manage change to attain the organization’s strategic ends and successful performance” (Stefl, 2008, p. 368) is becoming an important skill and is increas- ingly the focus of academic enquiry (Lega, Prenestini, & Spurgeon, 2013; Martin & Learmonth, 2012; Uhl-Bien & Marion, 2009). Although the terms are often used interchangeably, management and leadership are in reality two different concepts. Management is “how,” while leadership defines “what” and “why” a task is accomplished. Management tends focus on the status quo of the organization, its systems, supervision, and process procedures, while leadership is dynamic and is continuously searching for new processes to ensure the future of the organization (Lorden, Lin, & Cote, 2015). Management therefore needs to have leadership capa- city in order to run the organization, and a member of the organization can be a leader even if he or she does not formally fill a management role (Grady & Dickson, 2016).

The recent shift in attention away from management toward leadership reflects the current evolution in healthcare organizations aiming for higher levels of performance. Traditionally, heavy bureaucracy has characterized healthcare organizations, and upper levels of hierarchy have been more interested in planning and budgeting inputs than checking results (Kubica, 2007). This traditional approach however has often led to higher spending rather than improved outcomes, so many health systems around the world have implemen- ted “managerialization” by measuring outputs and shifting attention to achieved results (Kaplan & Porter, 2011, p. 46). This has taken place in Italy as in other countries, and from the early 1990s with Law D. Lgs. No. 502 of 1992 managerial tools such as budget- ing systems, accrual accounting, cost accounting, and performance indicators have been introduced for plan- ning development and better performance (Adinolfi, 2014; Borgonovi, 2004).

Although managerial practices, which include plan- ning, budgeting, organizing, staffing, and controlling (Kotter, 1996), are important for good clinical

performance (Bloom, Propper, Seiler, & van Reenen, 2009), good leadership is essential for them to be per- ceived as truly helpful in guiding personnel in day-to- day work, in creating consensus and commitment, and therefore for the success of the organization (VanVactor, 2012). Leadership is the critical success factor in building and developing processes leading to qualitative performance, and management tools are supporting elements for leading organizations (Bierman & Clark, 2007).

For this purpose, leadership needs to be present in all those involved in the health creation process, from strategic management, to middle management, and healthcare professionals. It is equally important for management tools to be present and implemented in operational processes supporting clinical functions. The role of the healthcare professional has also evolved. In the past, professional and cultural autonomy did not allow management interventions (Davies & Harrison, 2003). Today, clinical/critical pathway tools, process re- engineering approaches, performance assessment tools, and lean management techniques are part of health professionals’ working lives, and they themselves are required to develop managerial and leadership skills along with medical ones (Lin et al., 2011).

It is clear that leadership plays a decisive role in creating good-quality performance. Hospitals therefore need to generate conditions to develop and put in place leadership, with particular reference to four main areas: management support tools, definition of targets and processes, sharing and participation, and identification of critical aspects.

Regarding management support tools, leaders require tools to coordinate the work of personnel, and align behavior toward results, and monitor and evaluate performance. They need to enable better collaboration between professionals and supply background to prac- tices in service delivery and feedback on the quality and appropriateness of clinical processes (de Jong & Den Hartog, 2007). Managers and professionals who base their decisions on information from measurement tools are able to develop a continuous improvement process. In fact, there is evidence that the use of management support tools for leaders is correlated with better results in terms of efficiency, efficacy, and quality (Bloom et al., 2009; Mazzocato, Savage, Brommels, Aronsson, & Thor, 2010).

The second area of management where leadership is required, development and implementation of targets and processes, entails starting with targets and laying down clear and shared procedures. It is an extremely complex matter to reach high performance levels and great attention needs to be paid to targets and assigning

811 INTERNATIONAL JOURNAL OF PUBLIC ADMINISTRATION

responsibilities. In complex organizations like hospitals, it is crucial for goals to be shared by different manage- ment levels if positive outcomes are to be reached (Anantatmula, 2008). Leaders need to define priorities clearly, make everyone aware of them, and ensure that the efforts of everyone are aimed at reaching them (Duarte, Goodson, & Arnold, 2013; Manion, 2005).

In the third area, conditions need to be favorable because the effectiveness of managerial tools is closely linked to their acceptance as crucial factors for governance rather than simply bureaucratic procedures (Byrkjeflot & Kragh Jespersen, 2014; Veillard et al., 2005). It is not sufficient for management tools to be present; good per- formance requires them to be recognized, shared, and to be part of professional day-to-day activities. Clinical skills are the added value in health organizations for obtaining good results, but better outcomes and developing manage- ment culture in health professionals imply that profes- sionals need to be aware of the value of management support tools (Emanuel & Pearson, 2012).

The last area, identifying critical areas, highlights the need to verify possible difficulties in the management process of improving performance. This is closely linked to the other areas previously investigated. There can in fact be various causes for improvement being slowed or hindered, including the unsuitable nature of manage- ment tools, inconsistency among targets, tools, and results, and an excessive management emphasis on bureaucracy. The hospital needs to identify these critical elements and put in place correction measures.

Method

Research followed a structured methodology; the aim was to build a useful tool for studying the implication of leadership for quality improvement processes. The research was carried out in the Joint Commission Italian Network. This is made up by more than 30 Italian Healthcare Organizations, both hospitals and other organizations, and comprises all Italian JCI- accredited organizations and other important Italian organizations interested in JCI methodology.

The JCI methodology focuses on defining processes and behaviors to improve outcomes; working on pro- cesses to impact an outcome is a strong link to research question and is related to the quality performance improvement of healthcare organizations. The JCI approach to quality performance improvement starts from the idea that integration of all quality manage- ment and improvement activities throughout the orga- nization results in improved patient outcome (JCI, 2014).

JCI uses the concept of the standard; standards set uniform and high expectations for the safety and qual- ity of patient care, and country-specific considerations related to compliance with those expectations are part of the accreditation process. All standards contain statements of intent for each standard and measurable element for assessing compliance with it.

A survey questionnaire was developed to study lea- dership impact on quality improvement processes. It defines four macro-areas of investigation (Table 1).

The first macro-area covers response to questions about “How hospitals measure quality performance” and focuses on methods of measuring. It identifies the aspects measured and the type of measurement tools used. Research focuses on certain types of tool, such as protocols and indicators, which are indicated by the JCI and other international guidelines as being key elements.

The second macro-area, “How hospitals define goals and processes,” covers the ways in which hospitals define their objectives, how these are assigned to pro- fessionals with relation to their position in the hierar- chy, how hospitals ensure that hospital policy is implemented, and methods of reporting.

The third macro-area, “Manager and professional satisfaction with quality performance system implemen- ted in their organizations,” features professionals’ and managers’ perception of the tools in use. They are asked for their opinion on performance assessment tools and tools for measuring compliance with guidelines, and the reasons underpinning these perceptions are described.

The final macro-area, “Difficulties in the perfor- mance quality improvement process,” features critical aspects of the performance assessment system and investigates the level of difficulty inherent in each phase of quality improvement.

Table 1. Macro-areas of investigation. Macro-Area Target

How hospitals measure quality Which aspects of performance are performance usually measured, and which tools

hospitals use to measure their performance quality, with a particular focus on guidelines and indicators.

How hospitals define goals and How hospitals set their goals, how processes they assign them to the

professionals, how they ensure that policies are implemented, and the type of reporting used.

Manager and professional Satisfaction of professionals and satisfaction with quality managers with tools used in their performance system organization to measure implemented in their performance and processes, and the organization reasons for this perception.

Difficulties in the performance Difficulties in identifying elements quality improvement process as parts of quality improvement

process.

812 S. FANELLI ET AL.

Two questionnaires were drawn up, one for hospital management and one for hospital professionals. The questionnaire for hospital management (Chief Executive Officer, Chief Administrative Officer, and Medical Director), henceforth “managers,” is made up of 20 questions; the questionnaire for hospital profes- sionals (Chief of Department, Chief of Medical Unit, and Nursing Manager at Department and Unit level), henceforth “professionals,” is made up of 19 questions.

Questionnaires were inclusive and sent to all profes- sionals with any management responsibility working in hospitals. They investigated which tools they use and their perception of leadership impact on organization.

Each of the four areas identified were investigated with more than one question. All questions were identified in relation to the Fifth Edition of Joint Commission International Manual for Hospitals (JCI, 2014); all of them are in fact related to a particular area of the manual, the “Governance, Leadership and Direction” (GLD). The JCI approach to leadership in this particular area comes from the idea that leadership comes from many sources in a healthcare organization, including governing leaders (gov- ernance), leaders, and others who hold positions of leader- ship, responsibility, and trust.

Once all questions were defined, they were validated by sending it to all quality officers of JCI-accredited hospitals to check the content and the form of the questions. Feedback received was used in the evaluation of each question and to draw up possible further ques- tions to include on the questionnaire. All feedback was analyzed and all suggestions were evaluated and imple- mented; from this process, the final version of the questionnaires was defined.

The questionnaire thus comprises two sections. The first shows the background, age band, and hospital the respondent works for, although first names and sur- names are not shown so that they remain anonymous. The second section contains the questions.

Thirty organizations were invited to take part in the survey: 21 accepted. Each hospital identified contacts: the management and about 10 professionals among the nursing staff and doctors as well as other health profes- sionals. Overall the questionnaire was sent to 359 pro- fessionals and 69 managers of the hospitals selected.

Responses were collected over a four-month period (July 2015–October 2015). Data was analyzed using a statistics software and relationships between variables (correlations and associations) were studied using sui- table indicators and tests (Pearson index; Phi index; Test-F; ANOVA) according to the type of variable and using a significance level of 5%.

At the end of the period for questionnaire response, all data were collected.

Description of the sample

Analysis covered only hospitals with at least six respon- dents, at least two managers, and at least four health professionals. Of the 21 hospitals that agreed to take part in the survey, 18 met these criteria. The sample is thus representative and comprises 60% of hospitals in Italy, which are members of the JCI Italian Network. Of the 18 hospitals, 10 are in the public and eight in the private sector. Thirteen are JCI-accredited and five are not.

A total of 147 responses were received: 43 from management and 104 from professionals. Breaking up the replies by type of hospital, 94 were from public sector hospitals (25 managers and 69 professionals) and 53 from private sector hospitals (18 managers and 35 professionals). In terms of accreditation, 99 responses were obtained from JCI-accredited hospitals (32 managers and 67 professionals) and 48 from non- JCI-accredited hospitals (11 managers and 37 profes- sionals). See Table 2.

Results

Now results are presented in the four areas of investigation.

How hospitals measure performance quality

Performance quality evaluation is important for the hospitals in the sample; all of them measure quality in various aspects. These include the standardization of services, risk processes, timing, and quality of treat- ments. Data show that tools for measuring these aspects are present within the organizations. Reports are most frequently made about risk; over 50% of the respon- dents state that they make or receive reports on it. Forty percent of respondents use reports on matters relating to speed, efficacy, and efficiency of treatment. A more detailed cluster analysis shows that in private hospitals reports are made more frequently (45%) than in public hospitals (34%) and management appear more favorable (41%) to reports than professionals (37%). A preliminary finding is thus that quality per- formance evaluation is the focus of management sys- tems that use reports as a monitoring tool.

Table 2. Composition of the sample. Hospitals Population Managers Professionals

Public 10 94 25 69 Private 8 53 18 35 JCI accredited 13 99 32 67 Not JCI accredited 5 48 11 37 Total 18 147 43 104

813 INTERNATIONAL JOURNAL OF PUBLIC ADMINISTRATION

Several tools for evaluating results are used, particularly:

● Descriptive reports on the attainment of policy goals (departments, units, and staff assessments);

● Formal audits of behaviors actually put into practice;

● Various methods of reporting behavior inconsistent with policy, for example through Public Relations Offices, customer satisfaction surveys, etc.);

● Identification and systematic collection of indica- tors showing the degree to which targets are reached.

Most organizations use indicators as evaluation tools; almost 75% of respondents, 81% of managers, and 71% professionals state that they are used in their organization. It is interesting that mainly managers in the public sector (76%) use written reports. Managers in the private sector on the other hand prefer audits (66%). Especially in the private sector, professionals appear to use unofficial tip-offs or reports (82%) when identifying behavior inconsistent with expecta- tions. This is in line with the fact that professionals are more familiar with operational tools in their day-to- day professional lives.

In general, there is a strong tendency to measure results using the complete range of different tools. This is very positive as it reveals the ongoing effort being made to generate behavior aligned with expectations, and results consistent with the planned targets. About 80% of respondents stated they used concretely at least two tools, which shows that there is a widespread desire to emphasize consistency between the results and the policies pursued.

There is a level of criticality regarding reports on behavior compliance with guidelines and protocols. Only slightly more than 35% consider reporting ade- quate. Closer examination shows that it can be difficult to make use of guidelines and protocols in practice, probably because the professional culture is based on clinical processes and professional skills and teamwork. Here again there is however a significant difference between the public and private sectors; 49% of respon- dents from the private sector state that there is ade- quate presence of reports on measuring compliance, as against only 29% from the public sector.

In general, hospitals are strongly oriented toward performance measurements and both professionals and management use these. Indicators are tools used to measure a wide range of targets and results, but the

evaluation of consistency of applications of guidelines is not as frequent as it might be.

How hospitals define goals and processes

Measurement processes of results are based on policies pursued and targets identified. The method of identifi- cation of targets and processes makes it possible to evaluate the management process and the level of col- laboration in performance evaluation. Considering this issue more closely, it is useful to describe how hospital management assigns targets to chief medical depart- ments or units and subsequently to individual staff members.

Considering management behavior, a first observa- tion is that goals are usually formalized (93%) and are made clear and explicit. This shows that management supervision is well rooted in the sample. A second observation is that goals are extremely various and include the following: process standardization; risk reduction in critical management processes; clinical services delivered speedily and efficaciously, with effi- cient use of resources; and delivery of high-quality services using evidence-based practice.

The high number of different goals is an indicator that management is aware of the need to take into account multiple aspects of quality. This awareness has however a weakness; goals do not always corre- spond to systematic assessment systems. Reports in fact are not used in 60% of cases overall. They are used in 25% of cases for standardization targets; 72% of cases for risk reduction targets; 44% of cases for efficacious care; and 23% of cases for high-quality treat- ment targets.

This fact highlights an area of potential improve- ment; setting up a reporting system consistent with the system of defining targets. In reality, there appears to be a system in place where targets generate behavior expectations, which cannot be concretely evaluated because there are no robust reporting systems. This happens, for example, for targets linked to the standar- dization of processes, which are considered extremely important but where reporting systems are weak. This means that on the one hand organizations aim to modify behaviors, but then they are unable to measure behaviors systematically. Moreover, defining targets that are not systematically measured leads to tension in efforts to reach performance targets.

Turning to how leaders assign targets within units or departments, 75% identify targets formally. This is sig- nificant and shows that organization as well as clinical

814 S. FANELLI ET AL.

tools are taken seriously. It is however true that 17% assign targets informally, so a culture of informality persists to some extent in some cases. As for targets, 45% of respondents use targets referring to more than one aspect (target care levels and clinical risk); this confirms that formalizing targets is now part of health- care organizations.

As noted above, the use of reporting is still unsatisfac- tory, although regular reports are widespread in monitor- ing systems. About 70% of the sample receive only one report and the most frequent type of content (about 50% of the sample) is risk. The remaining 30% state that they receive more than one periodical report. Only one respon- dent states that no report at all is received. The emphasis on risk at the expense of other aspects of quality is a weakness. Only 23% of professionals interviewed state that they receive care quality reports, which are evidence-based.

Focusing on how targets are defined, it is necessary to clarify two distinct phenomena, especially for profes- sionals. There is widespread dissatisfaction with guide- line compliance and secondly there is dissatisfaction about reporting. About 50% are satisfied with the guidelines themselves, but only 33% consider the reporting system to be adequate. The situation among management is similar; just over 60% believe protocols and guidelines are adequate but only just over 40% believe the reporting system to be adequate.

This may reflect the different roles of managers and professionals. Management tend to find the presence of protocols and guidelines to be adequate, although note that about 30% of managers believe that there are too many of these tools in use. On the other hand, 30% of professionals state that tools such as protocols and guidelines require improvement in their hospital.

The key to establishing whether performance sys- tems are firmly rooted in an organization is to evaluate how they encourage using protocols, guidelines, and tools to monitor organizational behaviors.

Three clear points emerge. In the first place, it is clear that targets are not supported by incentives and overall are given little support by management; 14% of managers interviewed state that incentives are used for developing guidelines and monitoring related behavior, but this percentage is only 2% among professionals. In the second place, few of the respondents state that there are rules for encouraging the development of guidelines and monitoring behavior (25% of managers and 31% of professionals). Lastly, there is a strong emphasis on direct contacts (60% managers and 67% professionals).

There are also several critical issues for management in the assignment of quality goals. Targets on process standardization are used by 80% of managers. However, 60% state that they do not receive regular

reports on the attainment of targets. Fourteen percent of the sample state that they have defined no targets. Therefore, as shown in Table 3, only a quarter of managers interviewed use targets and reporting coherently.

To sum up, evaluation of quality performance in this context shows: a good level of formalization of targets, especially for quality of care and for clinical risk; reporting systems are not always robust enough for performance evaluation systems; partial attention to protocols and guidelines; and the widespread use of informal mechanisms supporting the use of protocols.

Managers and professionals belief in performance quality systems implemented in their organizations

In this area, one of the first aspects analyzed is the level of satisfaction with measurement systems of performance quality. Only 40% of the respondents state they are satis- fied with them, while about 60% say they are dissatisfied. These percentages are similar between clusters and there are no substantial differences between professionals and managers, or between the public and private sectors.

Research then looked for correlations between this level of satisfaction and three other variables: target defi- nition, perceived difficulties in measuring, and adequate reporting systems. The correlations however did not prove significant except for actions for evaluating profes- sionals (Pearson index 0.570; p-value 0.000). It appears that satisfaction in performance evaluation depends on specific aspects of local situations and is closely correlated with the evaluation of professionals. In other words, where there is satisfaction with the evaluation of profes- sionals, there is also satisfaction with performance eva- luation. This confirms the correlation of the role of professionals with performance quality; it depends not so much on tools, but rather on professionals’ capacity to integrate their actions and enhance their skills. This is an extremely important point for performance evaluation strategy. The capacity to evaluate quality of services is found directly correlated with the capacity to integrate professionals into the system and enhance their input. A strong correlation is always present (0.570; 0.000), and is higher among professionals (0.593; 0.000).

Table 3. Targets and reports on process standardization. Have you defined a formal target for

process standardization for your staff?

Yes No

Do you receive regular reports on Yes 23.26% 2.33% standardized processes? No 60.47% 13.95%

815 INTERNATIONAL JOURNAL OF PUBLIC ADMINISTRATION

It is of course necessary to identify the causes of poor capacity in monitoring performance. Data shows that every situation has particular weaknesses and pro- blems. There are no correlations between different aspects of dissatisfaction; statistical distribution does not point to any specific or generalized reason. Analysis of the hospital as an overall organization, focusing on weaknesses in different tools (measurement tools, evaluation systems, coordination criteria and pro- tocols, and guideline development tools), reveals that one who is dissatisfied with the evaluation system of his/her organization identifies more negative aspects than positive ones. This implies that there will be a specific set of problems in each case.

This study thus highlights that there are specific rather than generalized causes of dissatisfaction affecting all hos- pitals. There appear to be three overall categories of pro- blem. The first category has to do with tools, and is essentially the absence of reports, indicators, and informa- tional support in general. The second category of problem concerns behaviors, specifically the low levels of collabora- tion typical between different units of organization and/or colleagues. The third category of problem is organization processes, such as the absence of incentive schemes and target fixing.

The study thus highlights that the difference between dissatisfaction and satisfaction is the absence of tools, and the presence of unsound behavior models and organization processes. This is particularly important for management in deciding where to concentrate investments and make intervention for improvement. The data shows clearly a correlation between respondent dissatisfaction and the lack of tools and/or dissatisfaction with particular behaviors and policies. Respondents are divided into two groups accord- ing to their satisfaction on how their organization monitors quality performance: satisfied and dissatisfied, and searched for a relationship between their satisfaction/dis- satisfaction and presence of organization of technical tools, behaviors, and organizational processes (see above). Table 4 shows the percentage of absence of each category in the different clusters.

In this context, collaboration is also an important issue. It was investigated in two aspects: collaboration between organizational units and collaboration among healthcare professionals.

A good level of collaboration was found between orga- nizational units (average 3.75 on a scale from 1—mini- mum to 6—maximum collaboration). There is also similarity between points of view of different organiza- tional professionals (managers 3.93; professionals 3.67) and two types of hospital (public sector hospitals 3.60; private sector hospitals 4.02). It is undoubtedly very posi- tive that the different clusters show no marked differ- ences, especially in view of the fact that collaboration is an important component of good performance.

Overall evaluation of collaboration between profes- sionals is even more positive (average 4.18), although there are marked differences between clusters. Evaluation by professionals is very positive (4.40), but manager responses show some criticality (3.63) (p-value t-test 0.000). In other words, professionals represent themselves as a team where collaboration has a signifi- cant value. This culture is present in all hospitals (pub- lic sector hospitals 4.01; private sector hospitals 4.47) and clearly expresses a view very closely linked to specific professional characteristics.

There is also a very positive opinion on quality testing methods (average 3.86). Most responses express higher levels of satisfaction among professionals (4.05) than for managers (3.42) (p-value t-test 0.003). Moreover, on issues of collaboration, managers appear to show significantly less satisfaction.

Lastly, it is interesting that there is no correlation between satisfaction and time dedicated to quality. It is not therefore the “time” resource that explains high satisfaction. It is in fact more closely linked to the significant component of collaboration (Pearson index 0.421; p-value: 0.000).

Therefore, the situation shows good collaboration between organizational units and among professionals, and satisfactory levels of collaboration in quality evaluation.

Difficulties in the process of quality performance improvement

The final part of the study looks at difficulty in the processes of evaluation and development of quality performance.

Table 4. Relationship between level of satisfaction expressed by respondents and inadequacy of tools/behaviors. Satisfied Dissatisfied

Managers Professionals Total Manager Professionals Total

Tools Behaviors Organization Total

50.46% 50.76% 63.10% 55.09%

53.15% 49.39% 58.69% 53.45%

52.09% 49.93% 60.42% 54.10%

63.16% 59.33% 65.41% 65.11%

61.69% 64.86% 64.18% 65.17%

62.02% 63.64% 64.45% 65.16%

816 S. FANELLI ET AL.

Respondent perception is that translating policy into behaviors is not a critical factor. On a scale of 1–6, where 1 is the highest complexity of translating policy into action, the average response was 3.84. However, although the average is very positive, distribution varies over the differ- ent clusters. Managers perceive that it is simpler to translate policy into behaviors than professionals do; manager aver- age is 4.28 compared to professionals 3.65 (p-value t-test 0.005). There is a similar divergence in views between staff of private and public sector hospitals; the averages are 4.17 compared to 3.65 (p-value t-test 0.014).

The study also investigated the issue of professional skills and possible difficulties in evaluation processes and skills development.

The evaluation and development of professional skills give a more complex picture. Data shows that only 20% respondents find no difficulty in translating policy into behaviors. The figures are similar for pro- fessionals and managers, but there is a higher percen- tage of private sector staff who find no difficulty (28%) than public sector staff (13%).

Four types of problem are identified in implementing policies: technical difficulties in measuring, inadequate tools available, professional resistance to change, and dif- ferences between groups of staff in the hospital. Technical problems were found in 35% of the sample, and more managers than professionals find this problem (46% vs. 30%), with no significant difference between public and private sector staff. Inadequate tools are a problem for 28% interviewees and again this is more of a problem for man- agers than professionals (32% vs. 26%) and for public rather than private sector staff (33% vs. 19%). Resistance to eva- luation is a problem for 40% of interviewees, with no difference between managers and professionals, but public sector staff finds it more of a problem than private sector staff (43% vs. 32%). Differences between particular groups in hospitals is a problem for 30% of the interviewees, slightly more for professionals than for managers (32% vs. 26%) and slightly more for public than for private sector staff (32% vs. 26%). In addition, almost half of the respon- dents (44%) identify more than one of the issues listed above; more managers than professionals (56% vs. 40%) and more public sector than private sector staff (47% vs. 40%) state they encounter more than one of the problems.

Conclusion

This research reveals clearly that performance quality evaluation is a key element in health organizations. Explicit performance evaluation policies enable hospital managers to develop and implement performance

evaluation tools in order to make quality policies robust across the hospital.

Management and healthcare professionals agree that the assessment of professional skills is important for a hospital. This is a key fact and implies that the two groups need to be on the same side. The capacity to check performance quality is directly related to the capacity to integrate the two sides and enhance their different contributions.

The second important result confirms previous findings from the literature, that hospitals require complex control systems that can measure numerous aspects of perfor- mance. These are what can supply positive results over time, especially for hospitals in the public sector.

The third finding has implications for the manage- ment role in hospital administration on the one hand and professionals on the other. Involving both sides and their differing skills and responsibilities, direct intervention and controls on the ground are key ele- ments of management style. Direct relationships and interaction need to be nurtured.

Performance quality assessment is a constructive experi- ence for hospitals: reports and tools such as indicators are set up and used for monitoring purposes. They require a high level of management awareness and a sound structure.

As expected, managers and professionals have a similar view of performance evaluation, and profes- sionals particularly believe that collaboration is a key factor in achieving results. Managers who make most use of formal evaluation tools however believe that it is necessary to invest in direct relationships, while profes- sionals who have fewer formal tools at their disposal are more favorable toward standardization.

Hospital management and professionals manage their organizations with a high level of awareness, and show a high level of skill in pursuing day-to-day objec- tives. They are not easily distracted by single issues and have great capacity in defining priorities. This is parti- cularly significant for managers, who are found to evaluate their own skills positively; they evaluate them- selves as aware and able to pursue strategic aims steadily.

The ability to translate policy into day-to-day beha- vior is a key aspect of quality performance, as well as other types of performance. Management particularly consider themselves as able to follow strategic goals. Data shows that close collaboration between different parts of an organization makes it more feasible to pursue them in day-to-day activity. They also show that collaboration impacts positively on achieving good-quality performance.

817 INTERNATIONAL JOURNAL OF PUBLIC ADMINISTRATION

Another key question is the evaluation and enhance- ment of professional skills. As expected, this is clearly a determinant in itself, but what is particularly important is the link with the performance system evaluation. Those who have a good opinion of the hospital perfor- mance system tend to have a good opinion of actions taken to enhance professional clinical skills. Like colla- boration between departments, this is clearly a factor enabling a positive working environment, which is in itself is a determinant of achieving goals. This view is widely shared by management and it therefore appears to be advisable to accompany the performance evalua- tion of professionals with professional enhancement policies.

The use of formal tools needs to be developed, particu- larly through the wider use of more detailed reporting. This should improve response as well as raise levels of collabora- tion between units of an organization.

Secondly, the level of satisfaction of management and professionals with the evaluation system is directly linked to particular aspects of the organization. Hospital charac- teristics such as public or private sector, small or large size, a teaching hospital or not, give rise to varying requirements as well as problem areas. Thus, factors determining whether the evaluation system is adequate or inadequate reflect specific conditions in the organization and specific pro- blems. Management and leading professionals are affected by the characteristics of the organization where they work, so that evaluation systems are to an extent “personalized,” This research makes an important contribution to the field by identifying the conditions widely agreed among man- agement and healthcare professionals to be necessary for success. These include collaboration between departments, hospital policy to valorize professional skills and formal reporting systems, as well as informal interaction between the two sides.

To summarize, the impact of the evaluation system on professional behaviors is determined by the use of formal tools accompanied by the capacity to generate collaboration and enhancement of professional skills. These two closely linked aspects, professional enhancement along with emphasis on quality processes, are the key variables for management to take into account in improving hospital management.

Acknowledgments

The research was carried out without funding. The results were presented at the International Journal of Public Administration Symposium held on May 25–27, 2016 in Palermo, Italy. We would like to thank the participants in

the panel, as well as the anonymous reviewers, for their useful and important feedback.

References

Adinolfi, P. (2014). Barriers to reforming healthcare: The Italian case. Health Care Analysis, 22(1), 36–58. doi:10.1007/s10728-012-0209-0

Anantatmula, V. S. (2008). Leadership role in making effec- tive use of KM. VINE, 38(4), 445–460. doi:10.1108/ 03055720810917705

Azzone, G., Masella, C., & Bertelè, U. (1991). Design of perfor- mance measures for time-based companies. International Journal of Operations & Production Management, 11(3), 77–85. doi:10.1108/01443579110143412

Beattie, E., & Mackway-Jones, K. (2004). A Delphi study to identify performance indicators for emergeny medicine. Emergency Medicine Journal, 21(1), 47–50. doi:10.1136/ emj.2003.001123

Berwick, D. M., James, B., & Coye, M. J. (2003). Connections between quality measurement and improvement. Medical Care, 41(1 Suppl), I30–I38. doi:10.1097/00005650- 200301001-00004

Bierman, A. S., & Clark, J. P. (2007). Performance measure- ment and equity. BMJ: British Medical Journal, 334(7608), 1333–1334. doi:10.1136/bmj.39251.660127.AD

Bloom, N., Propper, C., Seiler, S., & van Reenen, J. (2009). Management practices in hospitals. New York, NY: Health, Econometrics and Data Group.

Borgonovi, E. (2004). Aziendalizzazione e governo clinico [Managerialization and clinical governance]. Mecosan, 49, 5–8.

Brown, M. G. (1996). Keeping score: Using the right metrics to drive world-class performance. New York, NY: Quality Resources.

Burstin, H., Leatherman, S., & Goldmann, D. (2016). The evolution of healthcare quality measurement in the United States. Journal of Internal Medicine, 279(2), 154– 159. doi:10.1111/joim.12471

Byrkjeflot, H., & Kragh Jespersen, P. (2014). Three concep- tualizations of hybrid management in hospitals. International Journal of Public Sector Management, 27(5), 441–458. doi:10.1108/IJPSM-12-2012-0162

Curtright, J. W., Stolp-Smith, S. C., & Edell, E. S. (2000). Strategic performance management: Development of a performance measurement system at the Mayo clinic. Journal of Healthcare Management, 45(1), 58–68.

Davies, H. T. O., & Harrison, S. (2003). Trends in doctor- management relationships. British Medical Journal, 326 (7390), 646–649. doi:10.1136/bmj.326.7390.646

de Jong, J. P. J., & Den Hartog, D. N. (2007). How leaders influence employees’ innovative behaviour. European Journal of Innovation Management, 10(1), 41–64. doi:10.1108/14601060710720546

de Korne, D. F., van Wijngaarden, J. D. H., Sol, K. J. C. A., Betz, R., Thomas, R. C., Schein, O. D., & Klazinga, N. S. (2012). Hospital benchmarking: Are US eye hospitals ready? Health Care Management Review, 37(2), 187–198. doi:10.1097/HMR.0b013e31822aa46d

818 S. FANELLI ET AL.

Donabedian, A. (1988). The quality of care: How can it be assessed? JAMA, 260(12), 1743–1748. doi:10.1001/ jama.1988.03410120089033

Drucker, P. (2012). Managing in the next society. New York, NY: Truman Talley Books.

Duarte, N. T., Goodson, J. R., & Arnold, E. W. (2013). Performance management excellence among the Malcolm Baldrige National Quality Award winners in health care. The Health Care Manager, 32(4), 346–358. doi:10.1097/ HCM.0b013e3182a9d704

Emanuel, E. J., & Pearson, S. D. (2012). Physician autonomy and health care reform. JAMA, 307(4), 367–368. doi:10.1001/jama.2012.19

Grady, C. M., & Dickson, G. (2016). Can complexity science inform physician leadership development? Leadership in Health Services, 29(3), 251–263. doi:10.1108/LHS-12- 2015-0042

Griffith, J. R. (2007). Improving preparation for senior man- agement in healthcare. Journal of Health Administration Education, 24(1), 11–32.

JCI, Joint Commission International. (2014). Joint Commission International accreditation standards for hos- pitals (5th ed.). Chicago, ILL: Joint Commission.

Kaplan, R. S., & Norton, D. P. (1992). The balanced score- card–measures that drive performance. Harvard Business Review, 70(1), 71–79.

Kaplan, R. S., & Porter, M. E. (2011). How to solve the cost crisis in health care. Harvard Business Review, 89(9), 46–52.

Keegan, D. P., Eiler, R. G., & Jones, C. R. (1989). Are your performance measures obsolete? Management Accounting, 70(12), 45–50.

Khaliq, A. A., Walston, S. L., & Thompson, D. M. (2007). Is chief executive officer turnover good for the hospital? The Health Care Manager, 26(4), 341–346. doi:10.1097/01. HCM.0000299252.83935.ed

Kotter, J. P. (1996). Leading change. Boston, MA: Harvard Business Press.

Kubica, A. J. (2007). Transitioning middle managers. How executives can ensure their success. Healthcare Executive, 23(2), 58–60.

Langlois, E. V., Ranson, M. K., Bärnighausen, T., Bosch- Capblanch, X., Daniels, K., El-Jardali, F., . . . Røttingen, J.- A. (2015). Advancing the field of health systems research synthesis. Systematic Reviews, 4(1), 1–7. doi:10.1186/ s13643-015-0080-9

Lega, F., Prenestini, A., & Spurgeon, P. (2013). Is management essential to improving the performance and sustainability of health care systems and organizations? A systematic review and a roadmap for future studies. Value in Health, 16(1), S46– S51. doi:10.1016/j.jval.2012.10.004

Lin, B. Y.-J., Hsu, C.-P. C., Juan, C.-W., Lin, -C.-C., Lin, H.-J., & Chen, J.-C. (2011). The role of leader behaviors in hospital- based emergency departments’ unit performance and employee work satisfaction. Social Science & Medicine, 72(2), 238–246. doi:10.1016/j.socscimed.2010.10.030

Lorden, A. L., Lin, S.-H., & Cote, M. J. (2015). Measures of success: The role of human factors in lean implementation in healthcare. Quality Control and Applied Statistics, 60(1), 127–130.

Lynch, R. L., & Cross, K. F. (1991). Measure up!: Yardsticks for continuous improvement. Cambridge, MA: Basil Blackweel Inc.

Manion, J. (2005). From management to leadership: Practical strategies for health care leaders (2nd ed.). San Francisco, CA: Jossey.

Martin, G. P., & Learmonth, M. (2012). A critical account of the rise and spread of “leadership”: The case of UK health- care. Social Science & Medicine, 74(3), 281–288. doi:10.1016/j.socscimed.2010.12.002

Martinez, J. A., & Martinez, L. (2010). Some insights on conceptualizing and measuring service quality. Journal of Retailing and Consumer Services, 17(1), 29–42. doi:10.1016/j.jretconser.2009.09.002

Mazzocato, P., Savage, C., Brommels, M., Aronsson, H., & Thor, J. (2010). Lean thinking in healthcare: A realist review of the literature. Quality and Safety in Health Care, 19(5), 376–382.

Morse, J. H., Koven, S. G., Mundt, C. J., & Gohmann, S. F. (2008). The Kentucky initiative in health services contract- ing: The search for contracting of outcomes measures. International Journal of Public Administration, 31(6), 639–653. doi:10.1080/01900690701640994

Neely, A., Adams, C., & Crowe, P. (2001). The performance prism in practice. Measuring Business Excellence, 5(2), 6– 13. doi:10.1108/13683040110385142

Nerenz, D., & Neil, N. (2001). Performance measures for health care systems. Center for Health Management Research. Retrieved from www.hret.org/hret/programs/ chmr/pls/pls10.html

Nwabueze, U., & Mileski, J. (2008). The three dimensions of quality service: The case of service quality gaps in the U.K. national health service? International Journal of Public Administration, 31(10–11), 1328–1353. doi:10.1080/ 01900690801973519

Quinn, R. E., & Rohrbaugh, J. (1981). A competing values approach to organizational effectiveness. Public Productivity Review, 5(2), 122–140. doi:10.2307/3380029

Stefl, M. E. (2008). Common competencies for all healthcare managers: The healthcare leadership alliance model. Journal of Healthcare Management, 53(6), 360–374.

Tangsoc, J. C., & Bautista, J. M. S. (2016). Hospital service quality assessment and analysis: A multi-perspective approach. DLSU Business & Economics Review, 26(1), 30–49.

Torok, H., Kotwal, S., Landis, R., Ozumba, U., Howell, E., & Wright, S. (2016). Providing feedback on clinical perfor- mance to hospitalists: Experience using a new metric tool to assess inpatient satisfaction with care from hospitalists. Journal of Continuing Education in the Health Professions, 36(1), 61–68. doi:10.1097/CEH.0000000000000060

Uhl-Bien, M., & Marion, R. (2009). Complexity leadership in bureaucratic forms of organizing: A meso model. The Leadership Quarterly, 20(4), 631–650. doi:10.1016/j. leaqua.2009.04.007

van Schoten, S., de Blok, C., Spreeuwenberg, P., Groenewegen, P., & Wagner, C. (2016). The EFQM Model as a framework for total quality management in healthcare: Results of a longitu- dinal quantitative study. International Journal of Operations & Production Management, 36(8), 901–922. doi:10.1108/IJOPM- 03-2015-0139

van Veen-Berkx, E., de Korne, D. F., Olivier, O. S., Bal, R. A., Kazemier, G., & Gunasekaran, A. (2016). Benchmarking operating room departments in the Netherlands: Evaluation of a benchmarking collaborative between eight university medical centres. Benchmarking:

819 INTERNATIONAL JOURNAL OF PUBLIC ADMINISTRATION

An International Journal, 23(5), 1171–1192. doi:10.1108/ Veillard, J., Champagne, F., Klazinga, N., Kazandjian, V., BIJ-04-2014-0035 Arah, O. A., & Guisset, A.-L. (2005). A performance

VanVactor, J. D. (2012). Collaborative leadership model in assessment framework for hospitals: The WHO regional the management of health care. Journal of Business office for Europe PATH project. International Journal for Research, 65(4), 555–561. doi:10.1016/j. Quality in Health Care, 17(6), 487–496. doi:10.1093/ jbusres.2011.02.021 intqhc/mzi072