Review on Energy Resilience

Contents lists available at ScienceDirect

Safety Science

journal homepage: www.elsevier.com/locate/safety

A resilience engineering perspective of safety performance measurement systems: A systematic literature review

Guillermina Andrea Peñalozaa,⁎, Tarcisio Abreu Saurinb, Carlos Torres Formosoa, Ivonne Andrade Herrerac

a NORIE/UFRGS (Built Environment Innovation Unit, Federal University of Rio Grande do Sul), Av. Osvaldo Aranha, 99, 3. andar, Porto Alegre, RS CEP 90035-190, Brazil b DEPROT/UFRGS (Industrial Engineering and Transportation Department, Federal University of Rio Grande do Sul), Av. Osvaldo Aranha, 99, 5. andar, Porto Alegre, RS CEP 90035-190, Brazil c Norwegian University of Science and Technology (NTNU), Department of Industrial Economics and Technology Management, NO-7491 Trondheim, Norway

A R T I C L E I N F O

Keywords: Safety performance Performance measurement systems Indicators Resilience engineering

A B S T R A C T

Although safety performance measurement systems (SPMSs) are key elements of safety management, previous research is usually limited to the proposal or assessment of indicators, without adopting a systems perspective. Furthermore, what counts as a well-designed SPMS is contingent to the adopted theoretical perspective, which is normally implicit. In this study, Resilience Engineering (RE) has been used for assessing SPMSs, providing an explicit and systems-oriented perspective. Previous research on SPMSs was analysed in a systematic literature review, with the aim of identifying whether RE offers a new perspective on SPMSs, and understand how RE has been put into practice in SPMSs. For each paper, there was an evaluation of how it accounted for five RE guidelines concerned with the design and implementation of SPMSs. The uptake of the guidelines was low on average, indicating that RE does not largely overlap with traditional assumptions of SPMS literature. However, there were several studies moderately or strongly aligned with those guidelines, suggesting that RE has been implicitly adopted to some extent. Descriptors were devised to convey approaches for the operationalization of the guidelines, providing a reference for the design of SPMSs informed by RE. A research agenda is also pro- posed.

1. Introduction

Performance measurement is a core element of managerial systems whatever the business dimension. It provides feedback from past and current performance, enables predictions to be made (Woods et al., 2015), and plays a role in satisfying the human psychological need for feeling in control (Dekker, 2014).

This study is concerned with safety performance measurement, addressed from a systems-oriented perspective. Instead of being limited to individual metrics, this investigation is concerned with Safety Performance Measurement Systems (SPMSs). A SPMS encompasses the design and selection of indicators; protocols for data collection, pro- cessing, and analysis; strategies for disseminating metrics; and a review process with the aim of regularly updating the system (Janicak, 2010). This systems-oriented perspective is relevant, as most studies are lim- ited to the selection or implementation of a set of isolated metrics,

which provide little insight on the nature of contributing factors to safety and may not be cost-effective (Øien et al., 2010).

SPMSs are either explicitly or implicitly based on a theoretical perspective on what is safety and how it can be obtained, which has implications for defining what counts as a relevant indicator and how they should be collected and analysed (Reiman and Pietikäinen, 2012). A common assumption adopted in the development and implementa- tion of SMPSs is that safety can be described, and therefore measured, in terms of a particular state or condition related to freedom from un- acceptable harm or risk (AHRQ, 2016). SPMSs that subscribe to this view rely mostly on lagging indicators, which monitor losses in terms of injuries and fatalities (Kjellén, 2009), being considered as reactive systems. This type of SPMS is widely used, as data collection and ana- lysis are relatively simple, metrics are easy to understand by both managers and workers, and these can be used for comparisons with other companies or national data (Sgourou et al., 2010). However,

https://doi.org/10.1016/j.ssci.2020.104864 Received 16 December 2019; Received in revised form 25 April 2020; Accepted 28 May 2020

⁎ Corresponding author. E-mail addresses: [email protected] (G.A. Peñaloza), [email protected] (T.A. Saurin), [email protected] (C.T. Formoso),

[email protected] (I.A. Herrera).

Safety Science 130 (2020) 104864

Available online 18 June 2020 0925-7535/ © 2020 Elsevier Ltd. All rights reserved.

T

reactive SPMSs have several drawbacks: (i) little predictive value, especially for accidents that arise from multiple contributing factors unlikely to reoccur in the same way; (ii) as there are relatively few serious accidents, lagging indicators make it difficult to distinguish trends from random variations; and (iii) their accuracy may be hindered by underreporting of undesired outcomes (Øien et al., 2011; Sinelnikov et al., 2015).

By contrast, other SPMSs give priority to leading indicators, which focus on the conditions or events that indicate the likelihood of out- comes (Kjellén, 2009), being considered as proactive. These SPMSs are concerned, for instance, with monitoring the status of the resources available for safe performance (Hallowell et al., 2013). However, pre- vious research has indicated that leading indicators tend to be more context-dependent than lagging indicators (e.g. accident rates), which makes it difficult to identify general proactive indicators that have strong predictive power (Lingard et al., 2017). Overall, it has been accepted that SPMSs should comprise a mix of reactive and proactive indicators (Herrera and Tinmannsvik, 2012) but these should not de- mand too much resources for implementation (Hale, 2009).

In this study, the lens of Resilience Engineering (RE) has been used to support the identification of strengths and weaknesses of SPMSs. RE is concerned with the observation, analysis, design and development of theories and tools to manage the adaptive ability of organizations in order to function effectively and safely (Nemeth and Herrera, 2015). In turn, resilience is “the expression of how people, alone or together, cope with everyday situations – large and small – by adjusting their perfor- mance to the conditions” (Hollnagel, 2017, p. 14). It means that an organization is resilient if it can function as required under both ex- pected and unexpected conditions. Performance adjustment implies in coping with the gap between work-as-done (WAD), which corresponds to what actually occurs in the workplace, and work-as-imagined (WAI), which corresponds to what people expect to occur (Hollnagel, 2014).

From the RE viewpoint, resilient systems have been defined as those that display four abilities at the system level, namely the abilities of monitoring, anticipating, responding, and learning (Hollnagel, 2017). This conveys a continuous improvement cycle, as learning must lead to the overall improvement of the other three abilities. Thus, it is rea- sonable to assume that RE is consistent with the aforementioned sys- tems-oriented perspective of performance measurement systems. Be- sides, RE acknowledges the need for both proactive (e.g. anticipating) and reactive (e.g. responding) safety management (Hollnagel, 2014), thus providing a framework for assessing SPMSs. In spite of this, the literature does not provide guidance on how to use the RE perspective for assessing SPMSs.

Thus, it is possible that previous empirical research studies on SPMSs have partly or intuitively adopted the RE perspective. An in- vestigation of the extent to which this occurs is necessary for two rea- sons: (i) it might help to clarify the extent to which RE premises, when applied to SPMSs, are new – this is relevant as there is still debate on whether RE offers an original perspective of safety management (Martinetti et al., 2019; Harvey et al., 2019); and (ii) it can shed light on how RE has been put into practice in the realm of SPMSs, even if un- intentionally – this has a pragmatic value for offering ideas to practi- tioners interested in designing SPMSs based on RE. Against this back- ground, the following research question was formulated for this research study: To what extent previous research on SPMSs have adopted the RE perspective?

This research question is addressed through a systematic literature review on SPMSs, which is relevant considering that performance measurement is one of the most common safety management research topics (Zhou et al., 2015). To our knowledge, this is the first literature review on SPMSs (either or not using the RE perspective), considering a systems-oriented perspective. In fact, there have been reviews on that topic focused on some industries, but limited to the definition of spe- cific safety performance metrics. For example, reviews on leading in- dicators used for controlling major hazards have been carried out in

maritime (Jalonen and Salmi, 2009), chemical (Bellamy and Sol, 2012) and process industries (Reiman and Pietikäinen, 2010; Swuste et al., 2016).

2. Research design

2.1. Research stages

This research work was divided into two main stages. The first stage was a traditional literature review focused on seminal authors, with the aim of defining a set of guidelines for the design of SPMSs aligned with RE. Those guidelines were used as a reference for analysing relevant research studies identified in the systematic literature review. This in- itial stage was based on nine book chapters (Hollnagel and Woods, 2006; Nemeth et al., 2008; Dekker et al. 2008; Hollnagel, 2009; Woods and Branlat, 2011; Hollnagel, 2014; Saurin et al., 2016; Hollnagel, 2017; Dekker, 2019) and three papers (Herrera and Hovden, 2008; Lay et al., 2015; Woods et al., 2015). These studies have been mostly de- veloped by leading RE authors, whose ideas have been known to widely influence other researchers.

Those 12 publications were subjected to a thematic analysis (Braun and Clarke, 2006), by examining the design guidelines proposed by different research studies. In fact, these guidelines were fairly con- sensual, even though designated by different terms, such as “learn from experience” (Dekker et al., 2008; Lay et al., 2015), “learn from normal work” (Nemeth et al., 2008; Saurin et al., 2016) or “learn from ev- eryday work” (Hollnagel, 2017). As a result, five RE guidelines relevant to SPMSs were devised (see Section 3.1).

The second stage of this investigation was the systematic literature review, which followed the steps suggested by Moher (2010): identifi- cation of papers; analysis; selection; and inclusion (Fig. 1). In the identification step, eight databases were considered: Web of Science, Scopus, Science Direct, American Society of Civil Engineers (ASCE), Taylor & Francis Online, Emerald Full Text and Google Scholar. These databases were queried between the 7th and 9th of June 2019 and the

A n

al ys

is In

cl u

si o

n Se

le cti

o n

Id en

ti fi

ca ti

o n

Systematic literature review on SPMSs

Records identified  through database

searching (n =128)

Records after duplicates removed (n = 103)

Records screened (n = 103)

Records excluded

(n = 8)

Full‐text articles assessed for eligibility

(n = 95)

Full‐text articles

excluded, with reasons

(n = 52)

Studies included for qualitative assessment

(n = 43)

Fig. 1. Steps of the systematic literature review.

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

2

results for each of them were downloaded in single batches on the same day. The search criteria encompassed the terms “safety performance” AND “measurement system” OR “safety indicators” OR “safety measures” OR “safety metrics” OR “assessment” OR “evaluation” in the title, ab- stract, and keywords. At the end of this step, 128 papers were exported to a reference manager, and, after the removal of 25 duplicates, 103 papers were identified. In the analysis step, eight papers were excluded according to two criteria: papers written in other languages than Eng- lish (2 records); and full content access denied (6 records). Concerning the selection step, the title and abstract of the 95 remaining papers were analysed according to one inclusion criteria, namely the use of an empirical approach that accounted for at least one of the following: (i) the proposition of frameworks, guidelines or indicators; and (ii) the practical implementation of indicators into an organization routine. No constraints were set on the nature of the research method, which could involve experimental analysis, case studies, or simulations, among others. Then, 52 papers were excluded because these were essentially theoretical, without support from empirical data or did not provide enough details of the empirical study. As a result, 43 papers were se- lected and included in the database.

2.2. Data analysis

The 43 papers were analysed according to three categories of in- formation: (i) bibliometric information; (ii) extent to which SPMSs ra- ther than only metrics were described; and (iii) alignment with the five RE guidelines. Regarding (iii) it is worth noting that the guidelines were not, with a few exceptions, explicitly mentioned in the papers. The analytical framework was therefore imposed on the studies as a heur- istic device (see Table 1).

Thus, the papers were fully read and excerpts of text related to the guidelines were identified and coded into descriptors. These descriptors correspond to examples of approaches for the operationalization of the guidelines. Initially, specific descriptors were developed for each paper. Then, after several rounds of revisiting these specific descriptors, they were grouped into more comprehensive descriptors, generic enough to be applicable to several studies.

For instance, specific descriptors for the guideline “the SPMS should support the monitoring of everyday variability” were originally coded, for some papers, as “the study proposed indicators based on standardized observation checklists for monitoring unsafe behaviours and conditions” or “indicators were selected according to the components of certified safety management system”. Next, these were grouped into a more general- izable descriptor, such as “a set of safety requirements (e.g. physical protections) are defined upfront as a basis for monitoring through in- dicators”. Each study was associated to only one generic descriptor for

each RE guideline. Fifteen descriptors emerged from data analysis and, for each of

these, a degree of conformance to the RE guidelines was established, as follows: strong alignment (score 2.0), moderate (1.0), weak (0.5), and no alignment (0.0). The aim of this scale is to facilitate discussion about the extent to which the guidelines were adopted in different studies. Also, the guidelines and the studies were ranked based on their average scores. Procedures described in this section were carried out primarily by the first two authors of this paper, who initially analysed half of the selected papers and developed their own Tables with descriptors. After that, a cross-check analysis was carried out to compare both codifica- tions and obtain a consensus.

3. Results

3.1. Resilience engineering guidelines for SPMSs

Table 2 presents the five guidelines and the studies from which they emerged. All studies provided insights into two or more guidelines.

Guideline 1 prescribes that SPMSs should support the monitoring of everyday variability. It emerged from studies acknowledging that, in complex socio-technical systems, performance variability is an in- evitable part of everyday work (Hollnagel and Woods, 2006) and often necessary for the production of required outputs (Lay et al., 2015). Therefore, the same variability sources usually play out both in acci- dents and everyday work. As the latter is much more frequent than the former, it offers more opportunities to understand the nature of varia- bility (Dekker et al., 2008; Hollnagel, 2017; Dekker, 2019).

In turn, guideline 2 states that SPMSs should provide feedback in real-time to those directly involved in the execution and supervision of production activities. This guideline also arises from the complexity of the systems monitored by SPMSs. Control in complex systems relies as much (or more) on feedback as on feedforward mechanisms (Hollnagel and Woods, 2006). Furthermore, complex systems are continuously evolving, and therefore the system status may never be exactly the same again (Nemeth et al., 2008). Therefore, it is important to shorten the time lag between data gathering, data analysis, and feedback, which is a basis for action-taking (Saurin et al., 2016). For the purpose of real-time feedback, a mix of direct and indirect sources of information tend to be useful. A simple everyday example of this mix can be observed when driving a car: each driver can control speed and other performance parameters by visualizing the cockpit dashboard (direct access to in- formation), while at the same time listening to news in the radio about traffic conditions (indirect access to information). Real-time feedback can also benefit from two other approaches: (i) a diverse group of analysts in terms of knowledge and skills, which can quickly identify

Table 1 Framework of data analysis.

Categories of data analysis Exemplar information searched in the selected papers

General categories Bibliometric information – Journal, year of publication, country where the empirical study was carried out, and industrial sector

SPMS perspective – Main activities of the SPMS accounted for by the study: design or select indicators; collect and analyse data; report and provide feedback; act on findings (Kaplan and Norton, 1996; Neely et al., 2005).

RE guidelines (1) SPMSs should support the monitoring of everyday variability – Indicators that provide information of work-as-done (2) SPMSs should provide feedback in real-time to those directly involved in the execution and supervision of production activities.

– Reporting mechanisms or indicators that provide real-time feedback. – Decentralization of data collection and dissemination

(3) SPMSs should facilitate learning from what goes well, in addition to what goes wrong

– Indicators of desired outcomes (e.g. safe behaviours) and undesired outcomes (e.g. near misses, accidents). – Organizational routines for sharing and discussing information from these indicators

(4) SPMSs should offer insights into the management of trade-offs between safety and other business dimensions

– Organizational routines or indicators that support decision-making for coping with trade-offs (e.g. number of times that the stop work authority is exercised) between safety and other business dimensions

(5) SPMSs should evolve due to the changing nature of complex socio-technical systems

– Changes or improvements made in SPMS with the aim to keep them up-to-date in a dynamic environment

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

3

early warnings of deteriorating performance (Lay et al., 2015); and (ii) information technology, such as the use of wearables that monitor en- vironmental, physiological and cognitive performance in real-time, triggering sensory warnings to workers (Dekker, 2019).

Guideline 3 conveys that SPMSs should facilitate learning from what goes well, in addition to what goes wrong (e.g. accidents). Hollnagel (2014) uses the term “what goes well” to refer to any situation in which there is presence of safety. In line with Hollnagel, an operational defi- nition of what is covered by an assessment of what goes well is pro- posed, as follows:

(i) The assessment of latent conditions, such as unsafe conditions or unsafe behaviours, considering how people or systems keep those conditions under control. If this assessment is limited to pin- pointing latent conditions as deviations from prescription, it is interpreted as the traditional focus on what goes wrong;

(ii) The assessment of safety management activities, such as safety inspections, risk assessments, and training, considering how these activities were carried out (e.g. whether training was limited to rule-following or included the development of adaptive skills). This assessment may be restricted to the counting of positive ac- tions (e.g. number of safety inspections carried out), but this may be less important in comparison to how it is carried out; and

(iii) Instantiations of problem-solving with desired outcomes.

Near misses have been excluded from the definition of what goes well as these imply a release of energy (Cambraia et al., 2010), thus posing an imminent danger. Therefore, the adopted dividing lines be- tween what goes well and what goes wrong are: (i) the existence of energy release; (ii) a focus on how energy release was prevented, in- stead of only counting deviations from work-as-imagined; and (iii) if counting is used, it should be focused on positive actions rather than negative – i.e. the larger-is-best type of metric.

As for guideline 4, it states that SPMSs should offer insights into the identification and management of trade-offs between safety and other business dimensions. Given the multiplicity of participants in complex systems and their potentially conflicting goals, trade-offs such as those between safety and efficiency tend to be ubiquitous (Woods and Branlat, 2011; Dekker, 2019). In practice, these trade-offs usually pend in favour of acute goals, such as efficiency, cost reduction, and fast delivery, instead of chronic goals (e.g. safety) (Woods, 2015). Hollnagel (2009) conveys this same idea as a generic trade-off between efficiency and thoroughness. Efficiency pressures push the system to the use of less and less resources, while concerns with thoroughness mean that the system should take the time to plan and understand how to produce the required outputs (Dekker, 2019).

Lastly, guideline 5 is concerned with the evolution of SPMSs. It means that, in order not to become stale, SPMSs should evolve due to the changing nature of complex socio-technical systems (Dekker, 2019). For instance, if a SPMS points out that the “margin of manoeuvre” of an operation is getting narrower, a greater frequency of monitoring the size of this margin is necessary (Woods, 2015). Guideline 5 can also be interpreted as a consequence of the law of requisite variety, which

indicates that the variety of the controller (e.g. SPMS) needs to match the variety of the controlled process (e.g. production processes mon- itored by a SPMS) (Ashby, 1991).

It is worth noting that these five RE guidelines are complementary, and, to some extent overlapping to other guidelines for SPMSs. For example, Peñaloza et al. (2020) compiled a set of general guidelines for SPMSs in several industrial domains – e.g. identify and prioritize critical processes to be covered by the SPMS. However, the five proposed guidelines have an explicit rationale based on RE, which justifies why these are relevant and under which circumstances this relevance grows – i.e. when complexity increases.

Furthermore, the proposed guidelines are logically associated with the four abilities of resilient systems proposed by Hollnagel (2017):

(i) Respond (knowing what to do): this is related to guideline 4, as the management of trade-offs implies making-decisions and re- sponding to them. It is also related to guideline 5, as the SPMS adaptation to a changing environment may be framed as an adaptive response;

(ii) Monitor (knowing what to look for): this is clearly connected to guidelines 1 and 2, which are directly concerned with monitoring;

(iii) Learn (being able to acquire the right lessons from the right ex- perience): this is directly connected to guideline 3.

(iv) Anticipate (knowing what to expect): this ability may be a result of the effective deployment of all guidelines, as this may result in the production of information for anticipating threats and opportu- nities in the short and long-term.

3.2. Bibliometric information

The 43 papers were published in 20 different journals, which sug- gest that a broad audience is interested in the topic of safety perfor- mance measurement. Safety Science had the largest number of papers (37%), followed by the Journal of Construction Engineering and Management (7%). Other well-known journals in the safety science field had low participation, such as Accident Analysis and Prevention, Journal of Safety Research, Reliability Engineering and System Safety, and Process Safety and Environmental Protection – each one of these accounted for 4.6% of the total.

The empirical studies reported in the papers were carried out in 19 countries, with a higher frequency of the United States (11 papers), Norway (5) and Australia (3) followed by Italy, Hong Kong, Poland, Slovenia, Spain and UK (2 each). In total, 82% of the studies were published in the last six years (from 2013 to 2019). Several industrial sectors were addressed, as shown in Fig. 2.

3.3. SPMS perspective: systems or indicators?

Fig. 3 shows the frequency in which the selected papers took into account the main activities involved in performance measurement systems (Kaplan and Norton, 1996; Neely et al., 2005), namely: design and/or select indicators; collect and analyse data; report and provide feedback; and act on findings. Fig. 3 was based on a thorough analysis

Table 2 RE guidelines for SPMSs.

Guidelines/sources [a] [b] [c] [d] [e] [f] [g] [h] [i] [j] [k] [l]

(1) SPMSs should support the monitoring of everyday variability x x x x x x x x x (2) SPMSs should provide feedback in real-time to those directly involved in the execution and supervision of

production activities x x x x x

(3) SPMSs should facilitate learning from what goes well, in addition to what goes wrong x x x x x x x x x (4) SPMSs should offer insights into the management of trade-offs between safety and other business dimensions x x x x x x x x x (5) SPMSs should evolve due to the changing nature of complex socio-technical systems x x x x x x x

[a] Hollnagel et al. (2006); [b] Herrera and Hovden (2008); [c] Nemeth et al. (2008); [d] Dekker et al. (2008); [e] Hollnagel (2009); [f] Woods and Branlat (2011); [g] Hollnagel (2014); [h] Woods (2015) [i] Lay et al. (2015); [j] Saurin et al. (2016); [k] Hollnagel (2017); [l] Dekker (2019).

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

4

of how each paper addressed those activities (see Appendix A). The focus on indicators instead of systems is made evident by the fact that the most frequent activities were the design or selection of indicators, and data collection and analysis (42%). Furthermore, in most studies there were long time lags (up to one year) between data collection, analysis, and feedback.

Only three studies (7%) have explored all activities involved in SPMSs (Cameron and Duff, 2007; Li et al., 2015; Awolusi and Marks, 2016). For example, Awolusi and Marks (2016) defined real-time in- dicators in close collaboration with project participants. Those re- sponsible for data collection received training for the identification of behaviours and conditions to be monitored. Immediate feedback was provided to construction workers so as they could adjust their perfor- mance. The implementation of corrective actions involved training on the avoidance of awkward postures during ground-level work.

3.4. Assessment of the RE guidelines

3.4.1. SPMSs should support the monitoring of everyday variability Fig. 4 presents the descriptors obtained for guideline 1

(mean = 0.70). The study by Raben et al. (2018) was the only con- nected to descriptor A. It provided a sound example of applying guideline 1, as it was based on a deep understanding of WAD as a whole, instead of disconnected fragments. In that investigation, the variability of everyday work of the blood sampling process was mod- elled through the Functional Resonance Analysis Method (FRAM). The proposed model was used to identify key inter-related functions that contribute to desired outcomes. Then, a set of candidate leading in- dicators were identified for those functions, such as “to identify patient with special needs”, “to walk to blood sampling room when called”.

As for descriptor B, it refers to studies concerned with the mon- itoring of everyday variability based on a set of safety requirements defined upfront as a reference for WAI. As the main drawback, the approach adopted by these studies implies in monitoring fragments of WAD as if these were independent from each other. Furthermore, safety

requirements are defined in a generic way. Although this makes it difficult to define what counts as a gap between WAI and WAD (e.g. there may be more demanding competences for the person collecting and analysing data), it provides flexibility and possibly a wide scope for monitoring everyday variability – this ambiguity is the main reason for considering descriptor B moderately aligned with guideline 1. For ex- ample, Laitinen et al. (2013) proposed a list of observable items as indicators of the behaviour of workers and conditions in manufacturing workplaces, such as “worker uses the necessary personal protective equipment, and does not take obvious risks”, and “workstation, tools and equipment are ergonomically designed”.

Descriptor C corresponds to relatively detailed specifications of WAI, which set a clear reference for the monitoring of everyday variability. Studies associated with descriptor 3 follow the traditional approach for safety inspections and audits, which is concerned with deviations from regulations, procedures and other manifestations of WAI. Although this facilitates the monitoring of the gap between WAI and WAD, there is an additional drawback in relation to studies asso- ciated with descriptor B: monitoring tends to ignore variability types that are not clearly related to WAI, as this is narrowly defined. Due to this drawback, studies related to descriptor C were coded as weakly aligned with guideline 1.

Some studies can be used to illustrate descriptor C. Podgórski (2015) in multiple sectors, Haas and Yorio (2016), in mining, and Ghahramani and Salminen (2019), in manufacturing, proposed metrics based on the safety management system components set by the stan- dard OHSAS 18001. The number of corrective actions completed, number of risk assessments, and number of safety training hours are examples of indicators suggested in those studies. It is worth noting that none of the three studies is concerned with the direct monitoring of the variability of production activities in which hazards play out. Rather, there is an emphasis on monitoring whether safety management ac- tivities were conducted as frequently as imagined.

3.4.2. SPMSs should provide feedback in real-time to those directly involved in the execution and supervision of production activities

Fig. 5 presents the descriptors obtained for guideline 2, which had the lowest average score (0.40), as a consequence of 30 out of 43 stu- dies not aligned with it. This suggests that, even when useful in- formation is produced by SPMSs, it might not be timely communicated to those who need that information most. This can be a consequence of an overreliance on centralized and bureaucratic safety management systems.

Descriptor A, the most aligned with guideline 2, derived from three

Fig. 2. Distribution of papers per sector.

Fig. 3. Percentage of SPMS activities accounted for by the selected studies.

Fig. 4. Descriptors of guideline 1 and their frequency. Notes: A: Context-spe- cific indicators were identified from a formal analysis and modelling of work- as-done; B: A set of safety requirements are defined upfront as a basis for monitoring through indicators. However, requirements are presented in a generic way (e.g. “use of personal protective equipment; safe behaviours”), thus also defining WAI in a generic way; C: A set of safety requirements (e.g. physical protections, management activities) are defined upfront as a basis for mon- itoring through indicators - e.g. “percentage of completed corrective actions in relation to safety audits”. The variability of everyday work is assessed against a strict definition of WAI; D: No alignment with the guideline.

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

5

studies (Li et al., 2015; Awolusi and Marks, 2016; Tamim et al., 2017) that provided real-time feedback to workers during the whole work shift. The indicators used in those studies were framed as “signs” or “early warnings” of safety risks in specific processes, in the gas and oil, and construction industries. In all three studies the support of in- formation technology was essential to provide real-time feedback. For example, Awolusi and Marks (2016) developed an automated mon- itoring tool to facilitate data collection and analysis of safety perfor- mance in a construction project. Activities were monitored through video cameras over a period of eight months. Observations were carried out by means of a checklist of safe and unsafe behaviours and condi- tions, from snapshots taken at one-hour intervals. The analysis and results were provided in real-time to workers, triggering appropriate interventions, such as adjustments in travel paths of heavy equipment to reduce proximity to workers. As a drawback, workers did not have direct access to information in order to self-organize, depending on external controllers.

Descriptor B was identified from 10 studies in which data were collected in real-time, but there was either a delay in data commu- nication or no feedback was provided to workers. For example, Rajendran (2012) performed more than one thousand observations of construction activities in a construction project, over a period of 37 weeks. Each observation lasted 10 s and unsafe behaviours of workers were identified. However, feedback was not provided to workers during or after observations – only managers received in- formation on the overall results of the observations.

3.4.3. SPMSs should facilitate learning from what goes well, in addition to what goes wrong

Fig. 6 presents the descriptors obtained for guideline 3, which had the highest average score (0.90). All studies were either coded as moderately or weakly aligned with this guideline. A common feature found in all papers is that these are concerned with the nature of the events that trigger learning (e.g. accidents), but do not discuss how and when organizational learning takes place. Also, no details are usually provided on the practical actions resulting from the use of indicators adopted to monitor these events. These were the main reasons why no study was coded as strongly aligned with guideline 3.

Descriptor A emerged from 29 studies that considered a mix of in- dicators addressing both what goes wrong and what goes well. However, this descriptor was considered to be moderately aligned with the guideline as the indicators based on what goes well were limited to safety management activities rather than production activities. For in- stance, Pawłowska (2015) investigated 60 companies in multiple sec- tors, concluding that the learning focus of safety indicators was de- termined by law provisions and certified safety management systems, for instance based on OHSAS 18001. As for what goes wrong,

companies had to present data required by regulations, such as the “cost of occupational accidents”. As for what goes well, there were metrics related to activities required for the certification of safety management systems, such as the frequency of safety inspections (Pawłowska, 2015).

Descriptor B was identified from two studies in which indicators provided insight into the underlying reasons of desired outcomes. For instance, Raben et al. (2018) modelled WAD in a blood sampling pro- cess, and identified key functions for achieving desired outcomes – e.g. “to identify patients with special needs”. Those authors suggested that the proposed model and those key functions could be used as a learning platform, for example, in training new staff. The other example of de- scriptor B is provided by Skogdalen et al. (2011), who analysed op- erators’ adaptability to risk for preventing deep water blowouts. Some indicators were devised from learning how operators identified early warnings of hazards getting out of control, and responded to these – e.g. if operators realized that the mud weight was too low, they took cor- rective actions.

Studies aligned with descriptor C were limited to facilitate learning from what went wrong in terms of accidents or other types of losses, such as machine breakdowns. For instance, Seyr and Muskulus (2016) proposed indicators for marine operations focused on failures during the installation and operation of an offshore wind farm – e.g. “annual failure rates for turbine subsystems”. Basso et al. (2004) combined in- cident investigations and analysis of major accidents of 50 companies from multiple sectors in order to define a threshold for “negative in- dicators” such as the “non-compliance with procedures about dan- gerous substances” and the “number of incidents due to wrong ob- servance of procedures and instructions”.

3.4.4. SPMSs should offer insight into the management of trade-offs between safety and other business dimensions

Fig. 7 presents the descriptors and results obtained for guideline 4 (mean = 0.70). Five studies proposed or applied tools that facilitated the management of trade-offs (descriptor A), having a strong alignment with guideline 4. One of these studies (Woods et al., 2015) proposed a framework for the analysis of whether the portfolio of indicators was balanced in terms of including indicators related to both efficiency and safety. Once a balanced portfolio is devised, it can facilitate the man- agement of trade-offs between safety and efficiency. The study by Rubio-Romero et al. (2018) was the only one that proposed indicators to directly assess the trade-off between safety and efficiency. These indicators were focused on the waste management sector and involved, for instance, the ability of employees to prioritise safety over produc- tion. However, little details on how to operationalize or implement

Fig. 5. Descriptors of guideline 2 and their frequency. Notes: A: Indicators re- sults are available in real-time to workers (e.g. observations of workers beha- viours, immediately followed by feedback from the observer to those being observed); B: Although indicators are gathered in real-time (e.g. observations), there is either a delay or no feedback is provided to workers. There can be feedback provided only to managers; C: No alignment with the guideline.

Fig. 6. Descriptors of guideline 3 and their frequency. Notes: A: There is a mix of indicators based on both what goes well and what goes wrong; B: The in- dicators only focus on what goes well – the implicit assumption is that learning results from understanding everyday work variability; C: The indicators only focus on what goes wrong – the implicit assumption is that learning results from understanding undesired outcomes.

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

6

these indicators were provided. Descriptor B encompasses 11 studies that, while not proposing in-

dicators or tools to monitor the trade-off, offer empirical evidence on how it plays out. This indirect approach of guideline 4 justifies the moderate, rather than strong alignment with it. For example, in the construction sector, Poh et al. (2018) provided quantitative evidence that the percentage of tasks completed was a good predictor of safety performance. Findings suggested that, from the perspective of the construction project as a whole, there was no trade-off between achieving safety and the expected project timeline (Poh et al., 2018).

Descriptor C was obtained from studies that have proposed in- dicators that indirectly shed light on the trade-off between safety and other business dimensions. No specific examples of managing the trade- off were provided by those studies. For example, Salas and Hallowell (2016) proposed indicators for the oil and gas sector, which offered insights into the trade-off between safety and efficiency – e.g. the “number of times that the stop work authority is exercised in the year”. A possible interpretation of this indicator is that the more the stop-work authority is applied, the more the trade-off is pending in favour of safety.

Also, for the oil and gas sector, Gerbec and Kontić (2017) proposed the joint analysis of a broad set of indicators, encompassing safety oc- currences, lost revenues, lost clients, fines, and implications for regional economy, among others. Those authors illustrate the relationships be- tween these indicators by reporting a case of spilling during methanol transhipment, which led to a major fire at the cargo terminal and had a strong business impact. An important message conveyed by that study is that both safety and financial losses usually go side-by-side.

3.4.5. SPMSs should evolve due to the changing nature of complex socio- technical systems

Fig. 8 presents the descriptors obtained for guideline 5 (mean = 0.70). As a drawback common to all studies, there was no longitudinal investigation describing how socio-technical systems evolved over a long time period, and how SPMSs coped with changes.

Descriptor A corresponds to nine studies that presented conceptual frameworks or models for updating the SPMS or set of indicators. Considering all five guidelines, this was the descriptor with the largest number of strongly aligned studies. For example, the framework de- veloped by Salas and Hallowell (2016) for oil and gas operations adopted a PDCA cycle for revising measures based on near-miss re- porting and safety performance outcomes. Haas and Yorio (2016) also used PDCA to identify relevant indicators for different production ac- tivities, which change over time. Leveson (2015) developed the System- Theoretic Accident Model and Processes (STAMP), which detects when the underlying assumptions of a leading indicator are no longer true, and therefore new or revised indicators are necessary. Sultana et al. (2019) also applied STAMP for the development of safety indicators in the oil and gas sector.

Raben et al. (2018) identified indicators from the blood sampling process based on a model of WAD (descriptor B). In principle, this model could be updated on a regular basis, supporting the update of corresponding indicators. However, no guidance on how to oper- ationalize this model was provided, which justifies the moderate alignment with guideline 5.

Descriptor C emerged from 24 papers, which pointed out the need for updating the SPMS, but do not provide any guidelines on how to do this – this justifies the weak alignment with guideline 5. For example, Skogdalen et al. (2011) refer to the need for monitoring organizational complexity that could lead to changes in safety management, including the SPMS (Janackovic et al., 2017). Hallowell et al. (2013) suggest that when an indicator does not lead to improvements, it should be removed from the SPMS. However, no further discussion on this type of assess- ment of indicators is provided.

4. Discussion

Fig. 9 summarizes the assessment of the guidelines. The level of alignment with RE was in general low, as indicated by the low average scores for the guidelines, ranging from 0.40 to 0.90 (in a scale from 0.0

Fig. 7. Descriptors of guideline 4 and their fre- quency. Notes: A: Tools or frameworks have been proposed to manage trade-offs between safety and other business dimensions; B: The study provides quantitative evidence that safety performance contributes significantly to performance in other business dimensions, such as quality and pro- ductivity; C: Some of the proposed indicators in- directly monitor the trade-off between safety and other business dimensions – e.g. stop work au- thority, requests of priority or emergency, cost and time delays; D: No alignment with the guideline.

Fig. 8. Descriptors of guideline 5 and their fre- quency. Notes: A: The study proposes a conceptual framework for updating SPMSs or set of indicators; B: If a SPMS is based on a model of WAD, it could be updated on a regular basis and then set a basis for updating the SPMS. However, the study does not show how this could be done; C: The study acknowledges the need for updating SPMSs, but provides no clear guidelines to do this; D: No alignment with the guideline.

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

7

to 2.0). On the one hand, low compliance with the guidelines points out that the RE perspective does not largely overlap with traditional as- sumptions of SPMS literature – i.e. it offers a new and under explored perspective. On the other hand, when considering studies individually, variability in the uptake of guidelines is observed.

The studies by Li et al. (2015) and Awolusi and Marks (2016) ob- tained the highest mean score (1.40), resulting from moderate align- ments with guidelines 1, 3 and 4, and strong alignments with guidelines 2 and 5. Furthermore, these two studies were sound examples of im- plementing all SPMS activities. A possible interpretation for this finding is that a system-based perspective for performance measurement can contribute to the adoption of RE, and vice versa. This may occur be- cause RE is systems-oriented, being concerned with the whole cycle of defining, collecting, and learning from metrics. It is also worth noting that these two studies were carried out in the construction industry, which has been rarely addressed in the RE literature, in comparison to other sectors, such as healthcare and aviation (Righi and Saurin, 2015).

In turn, the studies by Basso et al. (2004), in multiple sectors, and Coleman and Kerkering (2007) in underground coal mining, had the lowest mean score (0.20). Both of them had weak alignments with two guidelines and no alignment with the others. These studies are re- presentative of traditional approaches as the analysis of injuries and lost work-day rates are emphasized.

Fig. 9 also shows that there were examples of moderate or strong alignment with all guidelines. This points out that RE has been im- plicitly used by many studies on SPMSs. Therefore, a full uptake may be

facilitated by drawing upon existing strengths, as the guidelines are logically related to each other. Fig. 10 presents some important logical relationships between the guidelines, based on emerging insights from the reviewed papers. This figure highlights the central role played by guideline 1 “SPMSs should support monitoring of everyday variability”. In fact, information produced from this type of monitoring supports all other guidelines. For instance, learning from what goes well and what goes wrong (guideline 3) requires information on how variability is playing out. A similar reasoning applies to relationships with the other guidelines. In turn, guideline 1 may benefit from others, such as in the case of the feedback loop between guidelines 1 and 3. Learning from what goes well and what goes wrong can result in changing the ap- proach for monitoring everyday variability – e.g. learning can reveal that certain types of events or working situations are more worth monitoring than others, as they offer richer learning opportunities.

The wide implications of guideline 2 (feedback in real-time), which was the least adopted in the reviewed studies, are also shown in Fig. 10. In particular, real-time feedback is closely related to two other guide- lines that may involve real-time decision-making on the spot in pro- duction settings. Firstly, the management of trade-offs between safety and other business dimensions (guideline 4) might benefit from accu- rate real-time feedback on performance. Of course, real-time feedback may be less relevant when trade-offs are addressed by higher hier- archical ranks, involving decision-making at strategic and tactical le- vels. Secondly, real-time feedback can help to identify short-term monitoring and learning priorities (i.e. influencing guidelines 1 and 3),

Fig. 9. . Uptake of the guidelines by the reviewed studies. Notes: 1: SPMSs should support the mon- itoring of everyday variability; 2: SPMSs should provide real-time feedback; 3: SPMSs should facil- itate learning from what goes well, in addition to what goes wrong; 4: SPMSs should offer insights into the management of trade-offs between safety and other business dimensions; 5: SPMSs should evolve due to the changing nature of complex socio-technical systems.

Fig. 10. . Relationships between RE guidelines for SPMSs.

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

8

guiding the reallocation of SPMSs resources - e.g. people in charge of collecting and analysing data may focus on priorities identified from real-time feedback.

As for guideline 5, “adapting due to the changing nature of complex systems”, Fig. 10 suggests that it mostly depends on other guidelines. This make sense as this adaptation may be interpreted as a SPMS re- sponse to a changing environment: effective adaptation benefits from learning on what goes well and what goes wrong, and from monitoring everyday variability. Also, feedback in real-time may support SPMS short-term adaptive responses, such as the reallocation of resources.

5. Conclusions

5.1. Contributions of this study

This study has assessed to what extent previous research on SPMSs was aligned with the RE perspective. Forty-three research studies were analysed, considering five RE guidelines. Based on examples extracted from the literature, fifteen descriptors used for summarizing practical approaches emerged. Descriptors and corresponding studies strongly or moderately aligned with RE provide a reference for researchers and practitioners interested in designing SPMSs.

The number of studies associated with strong alignment with the guidelines was low, ranging from zero (SPMSs should facilitate learning from what goes well, in addition to what goes wrong) to nine (SPMSs should evolve due to the changing nature of complex socio-technical systems). This points out that: (i) RE perspective does not largely overlap with the assumptions of traditional SPMS literature; and (ii) RE is far from being mainstream in SPMSs research, despite offering a new perspective.

Some interactions between the five guidelines were identified, pointing out that these have synergistic relationships. Therefore, it might be possible to build upon strengths of SPMSs that are not RE oriented, if these partly account for the guidelines. In fact, the highest scoring studies identified in this review did not refer explicitly to RE, which suggests that there can be contextual factors that may naturally lead a SPMS to evolve towards that approach – e.g. opportunistic use of new technologies, and an organizational culture that values resilience.

5.2. Limitations

Some limitations of this study must be pointed out. First, there might be other RE guidelines or perspectives that are useful for SPMSs, such as the notion of “graceful extensibility” coined by Woods (2018). However, the set of five guidelines offered a robust account of RE, by being associated with the four abilities of resilient systems. Second, the real extent to which the guidelines were adopted may have been masked by the lack of implementation details in some papers – this is understandable as the papers were not intentionally concerned with the implementation of those guidelines. Third, the academic literature might not accurately represent the real diversity and approaches for SPMSs. Fourth, as usual in systematic literature reviews, the adopted inclusion and exclusion criteria may have missed useful studies. Fifth, while RE guidelines are theoretically sound, cause-effect links between their level of adoption and superior safety performance is elusive. This limitation must be put into perspective for two reasons: a SPMS is an element of a broader safety management system, and therefore the evaluation of its isolated impact on performance is difficult; and the same limitation applies to the non-RE oriented approaches for con- ceiving SPMSs.

5.3. Research agenda

The gaps in knowledge identified in the systematic literature review provided the basis for a research agenda. This agenda encompasses opportunities related to each individual guideline and for SPMSs as

whole, as follows:

(i) SPMSs should support the monitoring of everyday variability: there is a need for developing SPMSs concerned with monitoring WAD as a whole, rather than as a set of fragmented elements, as commonly addressed by checklists. This could benefit from the use of descriptions of WAD (e.g. by using FRAM or similar purpose methods) as a basis for conceiving and operating SPMSs. There is also a need for a deeper understanding of variability in safety management activities. Only the outcomes of these activities (e.g. number of risk assessments carried out), rather than their pro- cesses, are usually monitored by SPMSs;

(ii) SPMSs should provide real-time feedback: innovative data analy- tics and big data technologies offer a wide range of possibilities for expanding the use of this guideline (Poh et al., 2018; Melo and Costa, 2019). The effectiveness of these technologies for the pur- pose of real-time feedback might benefit from: criteria for prior- itizing activities in which real-time monitoring is cost-effective; the identification of different users of information and their require- ments in terms of contents and format; and the provision of or- ganizational resources (e.g. training, supervision) so as those re- ceiving real-time feedback can take immediate corrective actions;

(iii) SPMSs should facilitate learning from what goes well, in addition to what goes wrong: an initial research opportunity related to this guideline refers to the development of taxonomies of successful events. The expanded definition of what goes well that has been proposed in this paper (see Section 2) may be a starting point for a comprehensive taxonomy. Furthermore, there is a research gap related to the identification of barriers to combine learning from success and failure in the same organization – e.g. are there trade- offs between both approaches? Learning from what goes wrong is entrenched in regulations and safety management education. Thus, another necessary research contribution refers to more empirical evidence on the effectiveness and value added by learning from what goes well;

(iv) SPMSs should offer insights into the management of trade-offs between safety and other business dimensions: further SPMSs studies should focus on the explicit monitoring of trade-offs, and also on the reinterpretation of existing metrics from this perspec- tive. While there are established approaches of applying this guideline to production activities in which hazards play out (e.g. stop work authority), similar mechanisms could be devised for monitoring and managing these trade-offs in managerial activities that may create or amplify safety risks at the front-line - e.g. are there trade-offs between quality (or finance, or environment, etc.) management and safety?

(v) SPMSs should evolve due to the changing nature of complex socio- technical systems: further studies should explore longitudinal in- vestigations of SPMSs, shedding light on whether and how these evolve over time. It is also worth investigating the bidirectional relationship between SPMSs and safety performance – i.e. while trends in performance may require updates in the SPMS, it is also possible that changes in the SPMS affect performance.

Moreover, further studies are necessary to assess the utility and applicability of the guidelines and descriptors in the design or evalua- tion of SPMSs, considering different sectors, with distinct complexity characteristics. This may shed light on the extent to which context impacts on the relevance of each guideline. From a practical perspec- tive, the guidelines and descriptors could be used for the identification of improvement opportunities in real SPMSs, possibly making con- tributions towards making safety management systems more resilient. This line of inquiry may also set a basis for the development of a tax- onomy of maturity levels for SPSMs regarding RE.

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

9

Acknowledgements

This research received funding from the Coordination for the Improvement of Higher Education Personnel (CAPES), Norwegian

Agency for International Cooperation and Quality Enhancement in Higher Education (SIU/DIKU) and STERNA project. CAPES and the Council for Scientific and Technological Development (CNPq) provide scholarship to the first author.

Appendix A. Selected studies in light of SPMS activities and methodological approaches.

Studies SPMS stages

1. Design and/or se- lect indicators

2. Collect and analyse data 3. Report and provide feedback 4. Act on findings

Hallowell et al. (2013)

literature review, ex- pert panel, historical data

ND ND *R stop work authority program should be reiterated and stressed by the safety personnel. A lack of worker em- powerment may be a symptom of poor safety culture

Raben et al. (2- 018)

Functional Resonance Analysis Method (FRAM)

questionnaire, observation checklist

by means of FRAM (visual representation of a process)

ND

Woods et al. (- 2015)

historical data historical data, expert panel

over 3-years period, by means of the Q4-Balance framework

ND

Laitinen et al. (2013)

literature review observation checklist, cor- relation analysis, regres- sion analysis

over 3-years period, by means of graphical charts

ND

Awolusi and Marks (20- 16)

literature review observation checklist, his- torical data, computer- based, correlation analysis

feedback was provided in real-time during data collection, by means of graphical charts

specific training for ground-level work

Janackovic et a- l. (2017)

literature review, ex- pert panel, multicri- teria decision-making

ND ND ND

Pawłowska (2- 015)

literature review questionnaire, correlation analysis

ND ND

Skogdalen et al. (2011)

literature review, his- torical data

computer-based ND ND

Poh et al. (20- 18)

historical data computer-based, regres- sion analysis

ND ND

Rajendran (20- 12)

literature review, his- torical data

observation checklist, cor- relation analysis

over 1-year period, by means of graphical charts *R to be effective, a minimum of 30 observations per week are needed

Haas and Yorio (2016)

literature review, his- torical data, expert panel

questionnaire ND ND

Li et al. (2015) literature review, his- torical data

observation checklist, ex- pert panel, computer- based

feedback was provided in real-time during data collection. Then, results were reported over 7- month period, by means of graphical charts

specific safety training for critical unsafe behaviours (e.g. being struck by rebar falling from the crane hook)

Gerbec (2013) Resilience-based Early Warning Indicators (REWI)

historical data, question- naire

ND *R monitor process safety on a regular basis and develop procedures for safe operations compliant with work in- structions

Cameron and Duff (200- 7)

literature review, questionnaire

historical data, question- naire

over 6-month period, by means of graphical charts and reports

a greater dissemination of risk assessments in pre-start meetings improved subcontractor’s safety performance after 3-months intervention

Lingard et al. (2017)

historical data historical data, correlation analysis, regression ana- lysis

over 5-year period, by means of graphical charts *R management activities should be described as positive indicators (e.g. measures of actions taken to proactively manage workers’ safety)

Salas and Hall- owell (20- 16)

literature review, his- torical data

historical data, regression analysis

over 1-year period, by means of graphical charts *R process workflow or model is needed for establish safety indicators

Gopang et al. (2017)

literature review questionnaire, regression analysis

ND *R need improvements in safety measures, e.g., protective clothing, waste disposal system

Köper et al. (2- 009)

Balanced Scorecard historical data, question- naire, regression analysis

by means of Balanced Scorecard (visual repre- sentation of strategy map)

*R human resources strategy should be aligned with busi- ness strategy

DeArmond et a- l. (2011)

literature review questionnaire, historical data, correlation analysis

ND ND

Podgórski (20- 15)

literature review, multicriteria deci- sion-making

ND ND ND

Rubio-Romero et al. (201- 8)

literature review, ex- pert panel

questionnaire ND *R provide mechanisms to employees to have access to sources of help (e.g. prevention services, special installa- tions), so that they can deal with unexpected safety incidents

Tamim et al. (- 2017)

literature review, his- torical data

computer-based feedback was provided in real-time during data collection

ND

Hinze et al. (2- 013)

historical data historical data, correlation analysis

ND ND

Shea et al. (20- 16)

literature review questionnaire, correlation analysis

ND ND

Øien et al. (20- 11)

Resilience-based Early Warning Indicators (REWI)

historical data ND ND

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

10

Sadeghi et al. (2015)

literature review historical data ND *R determine the thresholds of the hazards tolerated by humans that are not available in standard documents (e.g. transmission mechanical energy)

Gerbec and Ko- ntić (2017)

Bayesian Belief Network

historical data ND ND

Di Gravio et al. (2015)

multicriteria deci- sion-making

historical data over 4-year period, by means of graphical charts *R promote a reporting culture and avoid missing infor- mation. The more the events database is accurate, the more the analysis will be flawless

Coleman and Kerkering (2007)

historical data historical data over 4-year period, by means of graphical charts *R improvements in underground mining technology by replacing hazardous techniques (e.g. the use of overshot muckers in small operations)

Sheehan et al. (2016)

literature review questionnaire, correlation analysis, regression ana- lysis

over 1-year period, by means of graphical charts *R develop a safety leadership training program for man- agers at all levels, especially for middle managers

López-Arquillos and Rubio- Romero (2- 015)

literature review, ex- pert panel

historical data ND ND

Robson et al. (- 2017)

historical data historical data, correlation analysis, regression ana- lysis

over 3-year period, by means of graphical charts and tables

*R decision-makers should not use audit scores as leading indicators in the absence of supporting empirical data

Sinelnikov et al. (2015)

literature review questionnaire, correlation analysis

ND ND

Guo and Yiu (- 2015)

literature review, questionnaire, System Dynamics

ND ND ND

Seyr and Musk- ulus (2016)

literature review, his- torical data

ND ND ND

Wong and Tse (2013)

literature review observation checklist over 1-year period, by means of graphical charts *R responsible for safety inspections should perform a set of sampling inspections of each item (e.g. 10). It should be documented for inspection training

Saurin et al. (- 2016)

literature review historical data ND *R safety reports should include key indicators from other performance areas (e.g. project time and cost as proxy measures of production pressures)

Leveson (2015) System-Theoretical Accident Model and Processes (STAMP)

historical data ND *R assumptions underlying engineering decisions (design rationale) should be documented (e.g. safety–critical changes) and used as leading indicators

Johnsen et al. (2013)

literature review, his- torical data, expert panel

ND ND ND

Sgourou et al. (2012)

historical data questionnaire monthly, by means of a conceptual model ND

Basso et al. (2- 004)

historical data historical data, correlation analysis

over 1-year period, by means of graphical charts ND

Ghahramani a- nd Salmin- en (2019)

literature review, his- torical data

historical data, regression analysis

ND ND

Sultana et al. (- 2019)

System-Theoretical Accident Model and Processes (STAMP)

historical data ND *R the plant should update indicators periodically and use threshold values as early warnings of critical items

Note: *R means that only recommendations for action-taking were presented. ND means that actual activity was not described.

References

Agency for Healthcare Research and Quality (AHRQ), 2016. PSNet Glossary. Patient Safety. Available in: https://psnet.ahrq.gov/glossary/p. (accessed June 6, 2019).

Ashby, W., 1991. Requisite Variety and its Implications for the Control of Complex Systems. Springer, Boston, MA, pp. 405–417.

Awolusi, I.G., Marks, E.D., 2016. Safety activity analysis framework to evaluate safety performance in construction. J. Constr. Eng. Manage. 143 (3), 05016022.

Basso, B., Carpegna, C., Dibitonto, C., Gaido, G., Robotto, A., Zonato, C., 2004. Reviewing the safety management system by incident investigation and performance indicators. J. Loss Prev. Proc. Ind. 17 (3), 225–231.

Bellamy, L. & Sol, V., 2012. A Literature Review on Safety Performance Indicators Supporting the Control of Major Hazards. National Institute for Public Health and the Environment, Ministry of Health, Welfare and Sport. RIVM rapport 620089001/ 2012.

Braun, V., Clarke, V., 2006. Using thematic analysis in psychology. Qualit. Res. Psychol. 3 (2), 77–101.

Cambraia, F.B., Saurin, T.A., Formoso, C.T., 2010. Identification, analysis and dis- semination of information on near misses: a case study in the construction industry. Saf. Sci. 48 (1), 91–99.

Cameron, I., Duff, R., 2007. Use of performance measurement and goal setting to improve construction managers' focus on health and safety. Constr. Manage. Econ. 25 (8), 869–881.

Coleman, P.J., Kerkering, J.C., 2007. Measuring mining safety with injury statistics: lost workdays as indicators of risk. J. Saf. Res. 38 (5), 523–533.

DeArmond, S., Smith, A.E., Wilson, C.L., Chen, P.Y., Cigularov, K.P., 2011. Individual

safety performance in the construction industry: development and validation of two short scales. Accid. Anal. Prev. 43 (3), 948–954.

Dekker, S., 2019. Foundations of Safety Science: A Century of Understanding Accidents and Disasters. Routledge.

Dekker, S., 2014. Safety Differently: Human Factors for a New Era. CRC Press. Dekker, S., Hollnagel, E., Woods, D. & Cook, R., 2008. Resilience Engineering: New di-

rections for measuring and maintaining safety in complex systems. Lund University School of Aviation.

Di Gravio, G., Mancini, M., Patriarca, R., Costantino, F., 2015. Overall safety performance of Air Traffic Management system: forecasting and monitoring. Saf. Sci. 72, 351–362.

Gerbec, M., Kontić, B., 2017. Safety related key performance indicators for securing long- term business development – a case study. Saf. Sci. 98, 77–88.

Gerbec, M., 2013. Supporting organizational learning by comparing activities and out- comes of the safety-management system. J. Loss Prev. Process Ind. 26 (6), 1113–1127.

Ghahramani, A., Salminen, S., 2019. Evaluating effectiveness of OHSAS 18001 on safety performance in manufacturing companies in Iran. Saf. Sci. 112, 206–212.

Gopang, M.A., Nebhwani, M., Khatri, A., Marri, H.B., 2017. An assessment of occupa- tional health and safety measures and performance of SMEs: An empirical in- vestigation. Saf. Sci. 93, 127–133.

Guo, B.H., Yiu, T.W., 2015. Developing leading indicators to monitor the safety condi- tions of construction projects. J. Manage. Eng. 32 (1), 04015016.

Haas, E.J., Yorio, P., 2016. Exploring the state of health and safety management system performance measurement in mining organizations. Saf. Sci. 83, 48–58.

Hale, A., 2009. Why safety performance indicators? Safety Sci. 47, 479–480. Hallowell, M.R., Hinze, J.W., Baud, K.C., Wehle, A., 2013. Proactive construction safety

control: measuring, monitoring, and responding to safety leading indicators. J.

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

11

Constr. Eng. Manage. 139 (10), 04013010. Harvey, E.J., Waterson, P., Dainty, A.R., 2019. Applying HRO and resilience engineering

to construction: barriers and opportunities. Saf. Sci. 117, 523–533. Herrera, I.A., Hovden, J., 2008. Leading indicators applied to maintenance in the fra-

mework of resilience engineering: a conceptual approach. In: Paper Presented at the 3rd Resilience Engineering Symposium, Vol. 28, pp. 30.

Herrera, I.A., Tinmannsvik, R.K., 2012. Key elements to avoid drifting out of the safety space. Proactive safety performance indicators.

Hinze, J., Thurman, S., Wehle, A., 2013. Leading indicators of construction safety per- formance. Saf. Sci. 51 (1), 23–28.

Hollnagel, E., Woods, D.D., Leveson, N. (Eds.), 2006. Resilience Engineering: Concepts and Precepts. Ashgate Publishing, Ltd..

Hollnagel, E., 2009. The ETTO Principle: Why Things that Go Right Sometimes Go Wrong. Ashgate, Farnham, UK.

Hollnagel, E., 2014. Safety-I and Safety-II: The Past and Future of Safety Management. CRC Press.

Hollnagel, E., 2017. Safety-II in Practice: Developing the Resilience Potentials. Routledge. Hollnagel, E., Woods, D.D., 2006. Resilience engineering: Concepts and precepts. Ashgate

Publishing Ltd. Jalonen, R., Salmi, K., 2009. Safety Performance Indicators for Maritime Safety

Management, Helsinki University of Technology, TKK-AM-9. Espoo, Finland. Janackovic, G., Stojiljkovic, E., Grozdanovic, M., 2017. Selection of key indicators for the

improvement of occupational safety system in electricity distribution companies. Saf. Sci.

Janicak, C.A., 2010. Safety Metrics: Tools and Techniques for Measuring Safety Performance, second ed. Government Institutes Press, Lanham, Maryland.

Johnsen, S.O., Okstad, E., Aas, A.L., Skramstad, T., 2013. Proactive indicators to control risks in operations of oil and gas fields. SPE Econ. Manage.. 4 (2), 90–105.

Kaplan, N., Norton, D.P., 1996. Using the balanced scorecard as a strategic management system. Harvard Bus. Rev.

Kjellén, U., 2009. The safety measurement problem revisited. Saf. Sci. 47, 486–489. Köper, B., Möller, K., Zwetsloot, G., 2009. The occupational safety and health scorecard-a

business case example for strategic management. Scand. J. Work Environ. Health 413–420.

Laitinen, H., Vuorinen, M., Simola, A., Yrjänheikki, E., 2013. Observation-based proactive OHS outcome indicators–validity of the Elmeri+ method. Saf. Sci. 54, 69–79.

Lay, E., Branlat, M., Woods, Z., 2015. A practitioner’s experiences operationalizing resi- lience engineering. Reliab. Eng. Syst. Saf. 141, 63–73.

Leveson, N., 2015. A systems approach to risk management through leading safety in- dicators. Reliab. Eng. Syst. Saf. 136, 17–34.

Li, H., Lu, M., Hsu, S.C., Gray, M., Huang, T., 2015. Proactive behavior-based safety management for construction safety improvement. Saf. Sci. 75, 107–117.

Lingard, H., Hallowell, M., Salas, R., Pirzadeh, P., 2017. Leading or lagging? Temporal analysis of safety indicators on a large infrastructure construction project. Saf. Sci. 91, 206–220.

López-Arquillos, A., Rubio-Romero, J.C., 2015. Proposed indicators of prevention through design in construction projects. Revista de la Construcción 14 (2), 58–64.

Martinetti, A., Chatzimichailidou, M.M., Maida, L., van Dongen, L., 2019. Safety I-II, resilience and antifragility engineering: a debate explained through an accident oc- curring on a mobile elevating work platform. Int. J. Occupat. Safety Ergon. 25 (1), 66–75.

Melo, R.R.S., Costa, D.B., 2019. Integrating resilience engineering and UAS technology into construction safety planning and control. Eng. Constr. Archit. Manage.

Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., 2010. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int. J. Surg. 8 (5), 336–341.

Neely, A., Gregory, M., Platts, K., 2005. Performance measurement system design: a lit- erature review and research agenda. Int. J. Operat. Prod. Manage. 25 (12), 1228–1263.

Nemeth, C.P., Herrera, I., 2015. Building change: Resilience Engineering after ten years. Nemeth, C., Wears, R., Woods, D., Hollnagel, E. & Cook, R., 2008. Minding the gaps:

creating resilience in health care. In: Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 3: Performance and Tools). Agency for Healthcare Research and Quality (US).

Øien, K., Massaiu, S., Tinmannsvik, R. K. & Størseth, F., 2010. Development of early warning indicators based on resilience engineering. In: Submitted to PSAM10, International Probabilistic Safety Assessment and Management Conference, pp. 7–11.

Øien, K., Utne, I.B., Herrera, I.A., 2011. Building safety indicators: Part 1–theoretical foundation. Saf. Sci. 49 (2), 148–161.

Pawłowska, Z., 2015. Using lagging and leading indicators for the evaluation of occu- pational safety and health performance in industry. Int. J. Occupant. Safety Ergon. 21 (3), 284–290.

Peñaloza, G.A., Saurin, T.A., Formoso, C.T., 2020. Monitoring complexity and resilience in construction projects: the contribution of safety performance measurement

systems. Appl. Ergon. 82, 102978. Podgórski, D., 2015. Measuring operational performance of OSH management system–A

demonstration of AHP-based selection of leading key performance indicators. Saf. Sci. 73, 146–166.

Poh, C.Q., Ubeynarayana, C.U., Goh, Y.M., 2018. Safety leading indicators for construc- tion sites: a machine learning approach. Autom. Constr. 93, 375–386.

Raben, D.C., Bogh, S.B., Viskum, B., Mikkelsen, K.L., Hollnagel, E., 2018. Learn from what goes right: a demonstration of a new systematic method for identification of leading indicators in healthcare. Reliab. Eng. Syst. Saf. 169, 187–198.

Rajendran, S., 2012. Enhancing construction worker safety performance using leading indicators. Pract. Period. Struct. Des. Constr. 18 (1), 45–51.

Reiman, T., Pietikäinen, E., 2010. Indicators of safety culture-selection and utilization of leading safety performance indicators (No. SSM–2010-07). Swedish Radiation Safety Authority.

Reiman, T., Pietikäinen, E., 2012. Leading indicators of system safety–monitoring and driving the organizational safety potential. Saf. Sci. 50 (10), 1993–2000.

Righi, A.W., Saurin, T.A., 2015. Complex socio-technical systems: characterization and management guidelines. Appl. Ergon. 50, 19–30.

Robson, L.S., Ibrahim, S., Hogg-Johnson, S., Steenstra, I.A., Van Eerd, D., Amick III, B.C., 2017. Developing leading indicators from OHS management audit data: determining the measurement properties of audit data from the field. J. Saf. Res. 61, 93–103.

Rubio-Romero, J.C., del Carmen Pardo-Ferreira, M., De la Varga-Salto, J., Galindo-Reyes, F., 2018. Composite leading indicator to assess the resilience engineering in occu- pational health & safety in municipal solid waste management companies. Saf. Sci. 108, 161–172.

Sadeghi, L., Mathieu, L., Tricot, N., Al Bassit, L., 2015. Developing a safety indicator to measure the safety level during design for safety. Saf. Sci. 80, 252–263.

Salas, R., Hallowell, M., 2016. Predictive validity of safety leading indicators: empirical assessment in the oil and gas sector. J. Constr. Eng. Manage. 142 (10), 4016052.

Saurin, T.A., Formoso, C.T., Famá, C.C., 2016. Criteria for assessing safety performance measurement systems: insights from resilience engineering. In: Resilience Engineering in Practice, Vol. 2. CRC Press, pp. 91–106.

Seyr, H., Muskulus, M., 2016. Safety indicators for the marine operations in the in- stallation and operating phase of an offshore wind farm. Energy Procedia 94, 72–81.

Sgourou, E., Katsakiori, P., Goutsos, S., Manatakis, E., 2010. Assessment of selected safety performance evaluation methods in regards to their conceptual, methodological and practical characteristics. Saf. Sci. 48 (8), 1019–1025.

Sgourou, E., Katsakiori, P., Papaioannou, I., Goutsos, S., Adamides, E., 2012. Using soft systems methodology as a systemic approach to safety performance evaluation. Procedia Eng. 45, 185–193.

Shea, T., De Cieri, H., Donohue, R., Cooper, B., Sheehan, C., 2016. Leading indicators of occupational health and safety: an employee and workplace level validation study. Saf. Sci. 85, 293–304.

Sheehan, C., Donohue, R., Shea, T., Cooper, B., De Cieri, H., 2016. Leading and lagging indicators of occupational health and safety: the moderating role of safety leadership. Accid. Anal. Prev. 92, 130–138.

Sinelnikov, S., Inouye, J., Kerper, S., 2015. Using leading indicators to measure occu- pational health and safety performance. Saf. Sci. 72, 240–248.

Skogdalen, J.E., Utne, I.B., Vinnem, J.E., 2011. Developing safety indicators for pre- venting offshore oil and gas deepwater drilling blowouts. Saf. Sci. 49 (8–9), 1187–1199.

Sultana, S., Andersen, B.S., Haugen, S., 2019. Identifying safety indicators for safety performance measurement using a system engineering approach. Process Saf. Environ. Prot.

Swuste, P., Theunissen, J., Schmitz, P., Reniers, G., Blokland, P., 2016. Process safety indicators, a review of literature. J. Loss Prev. Process Ind. 40, 162–173.

Tamim, N., Laboureur, D.M., Mentzer, R.A., Hasan, A.R., Mannan, M.S., 2017. A frame- work for developing leading indicators for offshore drillwell blowout incidents. Process Saf. Environ. Prot. 106, 256–262.

Wong, F.K., Tse, R.O., 2001. An alternative safety performance-monitoring tool: the safety-sampling method. HKIE Trans. 8 (1), 39–42.

Woods, D.D., Branlat, M., 2011. Basic patterns in how adaptive systems fail. Resilience Eng. Pract. 127–144.

Woods, D.D., 2015. Four concepts for resilience and the implications for the future of resilience engineering. Reliab. Eng. Syst. Saf. 141, 5–9.

Woods, D.D., Branlat, M., Herrera, I., Woltjer, R., 2015. Where is the organization looking in order to be proactive about safety? A framework for revealing whether it is mostly looking back, also looking forward or simply looking away. J. Contingencies Crisis Manage. 23 (2), 97–105.

Woods, D.D., 2018. The theory of graceful extensibility: basic rules that govern adaptive systems. Environ. Syst. Decis. 38 (4), 433–457.

Zhou, Z., Goh, Y.M., Li, Q., 2015. Overview and analysis of safety management studies in the construction industry. Saf. Sci. 72, 337–350.

G.A. Peñaloza, et al. Safety Science 130 (2020) 104864

12