Article Summary Writing using APA citation

RESEARCH Open Access

Evaluation frameworks for digital nursing technologies: analysis, assessment, and guidance. An overview of the literature Tobias Krick1,2

Abstract

Background: The evaluation of digital nursing technologies (DNT) plays a major role in gaining knowledge about certain aspects of a technology such as acceptance, effectiveness, or efficiency. Evaluation frameworks can help to classify the success or failure of a DNT or to further develop the technology. In general, there are many different evaluation frameworks in the literature that provide overviews of a wide variety of aspects, which makes this a highly diverse field and raises the question how to select a suitable framework. The aim of this article is to provide orientation in the field of comprehensive evaluation frameworks that can be applied to the field of DNT and to conduct a detailed analysis and assessment of these frameworks to guide field researchers.

Methods: This overview was conducted using a three-component search process to identify relevant frameworks. These components were (1) a systematized literature search in PubMed; (2) a narrative review and (3) expert consultations. Data relating to the frameworks’ evaluation areas, purpose, perspectives, and success definitions were extracted. Quality criteria were developed in an expert workshop and a strength and weakness assessment was carried out.

Results: Eighteen relevant comprehensive evaluation frameworks for DNT were identified. Nine overarching evaluation areas, seven categories of purposes, five evaluation perspectives and three categories of success definitions could be identified. Eleven quality criteria for the strengths and weaknesses of DNT-related evaluation frameworks were developed and the included frameworks were assessed against them.

Conclusion: Evaluators can use the concise information and quality criteria of this article as a starting point to select and apply appropriate DNT evaluation frameworks for their research projects or to assess the quality of an evaluation framework for DNT, as well as a basis for exploring the questions raised in this article. Future research could address gaps and weaknesses in existing evaluation frameworks, which could improve the quality of future DNT evaluations.

Keywords: Framework, Evaluation, Digital, Technology, Care, Nursing

© The Author(s). 2021 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Correspondence: [email protected] 1University of Bremen, SOCIUM Research Center on Inequality and Social Policy, Mary-Somerville-Straße 3, 28359 Bremen, Germany 2University of Bremen, High-profile Area of Health Sciences, Bremen, Germany

Krick BMC Nursing (2021) 20:146 https://doi.org/10.1186/s12912-021-00654-8

Background A large number of digital nursing technologies (DNTs) are currently being developed and tested in nursing practice [1, 2]. These technologies offer promising op- portunities to address existing societal challenges such as the shortage of skilled workers or the increasing de- mand for long-term care [3]. This article refers to digital nursing technology (DNT)

as defined by Krick et al. 2019 [2] and Krick et al. 2020 [4]. DNT are technologies that fulfil one or all of the fol- lowing criteria: i) “support the immediate action of a caregiver”; or ii) “contribute to the self-reliance of the person in need of care in such a way that direct on-site care assistance can be avoided”; or iii) “substitute the nursing support by using technology”, or iv) “support the training or education of nurses” [4]. The focal points of this article are the aspects i-iii. DNT’s can, for ex- ample, be information and communication technologies, robots, sensors, monitoring technologies, assistive de- vices, ambient assisted living technologies, virtual reality or tracking technologies [1, 2]. Professional nurses point to the need for improved

technological support in direct care to reduce physical strain and psychological stress [5], which e.g. could have a long-term impact on retention in the profession. On the other hand, use and acceptance in actual clinical support appears to be rather low [6, 7]. The reasons for the lack of acceptance and usage can be

very diverse, since DNT are complex interventions [8]. Specific reasons for non-adoption can be that technologies are not user-friendly (low usability) [5] or have no obvious perceptible benefit for actual work practice (job relevance, perceived usefulness) [5]. Privacy issues or cost concerns may also bea major concern for persons in need of care [9]. Scientific evaluations that provide information on technologies from different perspectives and viewpoints could help us to understand the bigger picture of DNT success and provide important insights on specific impact factors. Evaluation results can, for example, help decision- makers to facilitate the process of system implementation [10]. Evaluation conducted during the development process also has the potential to prevent system failures and misdevelopments [10]. Comprehensive evaluation frameworks that clearly

present important aspects of evaluation play a significant role in supporting researchers, decisionmakers and de- velopers in this process. Evaluation frameworks can be used to provide a structure for the evaluation of DNT as they provide information and definitions of technology success, evaluation areas, methods, and tools. In this way, they “facilitate a systematic approach” [11] in DNT evaluation. The information provided by evaluation frameworks can enable different stakeholders to gain a common understanding of the evaluation process and

help identify and decide on appropriate objectives and methods. This can help mediate the fit between research and practice-oriented approaches to evaluation [11]. This article focusses on comprehensive evaluation

frameworks that include information on multiple evalu- ation areas. There is a wide variety of frameworks in the literature and most of them are highly heterogeneous in content, approaches, and methodologies. Identifying and deciding on the appropriate evaluation framework can be very challenging, as there is no suitable overview of evaluation frameworks in the field of DNT available in the literature. It is also difficult to see how these frame- works differ in terms of purpose, and areas of evaluation, and their definition of success. Orientation must be pro- vided on these possibilities, and this overview was con- ducted to this end.

Objective and research question The objectives of this article are to provide orientation on existing comprehensive evaluation frameworks that can be applied in the field of DNT and to conduct a de- tailed analysis and assessment of these frameworks. The aim is to contribute to the discussion and understanding of what constitutes a good (DNT) evaluation framework and to offer field researchers guidance in the selection and application of evaluation frameworks. This article is thus guided by the following research

questions: (i) Which comprehensive evaluation frame- works that can be applied to DNT evaluation are available in the literature? (ii) What purposes, perspectives, and def- initions of success are described in these frameworks? (iii) What are the strengths and weaknesses of the included frameworks? (iv) Which areas of evaluation are repre- sented in the frameworks and where are the most overlaps and differences between these frameworks?

Methods Search process This overview [12] was conducted using three different search components (roman numerals). The goal was to identify frameworks of relevance to the nursing context according to the inclusion and exclusion criteria de- scribed further below. The search components com- prised: (i) a systematized literature search conducted in PubMed; (ii) a narrative review was carried out by searching google scholar, screening already identified lit- erature [4] and reference lists of the systematically iden- tified articles; (iii) information on relevant frameworks collected from four experts in the field of evaluation from the German “Cluster Zukunft der Pflege” (Future of Care Cluster) [13]. In this joint project, regular evalu- ations of digital nursing technologies have been and are carried out over the period 2017–2022. The search terms of the systematized literature search can be found

Krick BMC Nursing (2021) 20:146 Page 2 of 19

in Table 1. The complete search strategy, including all three components, is shown in Fig. 1.

Eligibility criteria for systematic search Scientific papers included in the search had to have been published between 2005 and 2020 in English. A sum- mary and full text had to be available. All information was gathered in March 2020, which limits the inclusion period from January 2005 to March 2020. The studies in question had to (i) either apply an

evaluation framework to a (nursing) technology or de- scribe the development of an evaluation framework. The selected frameworks had to (ii) be at least based on ei- ther a literature study, an empirical evaluation, or an ex- pert survey. They had to (iii) be directly related to technologies in (nursing) care, or comparable fields like the evaluation of digital health applications in general, but they had to be potentially suitable for the evaluation of DNT. (iv) The technology specific focus must lie on Information and Communication Technologies (ICT), telemedicine, telecare, sensor technology or robotics (or their sub-sectors), as these are the most common tech- nology areas in nursing care (definitions by Krick et al. 2019) [2]. (v) The frameworks could refer to the evalu- ation of different stages of the life cycle of a technology [14] e.g. before, during or after implementation. Exclusion criteria were: articles (i) focussing on frame-

works for the evaluation of specific medical technology fields (e.g. radiology, surgery) with no relation to nurs- ing; (ii) from developing countries or underdeveloped health systems; (iii) that described study protocols; (iv) with overviews that only present categorical systems without creating a framework; (v) focussing on frame- works related to technologies for education or training; (vi) focussing on fitness applications, wellness applica- tions or applications for general disease prevention; (vii) in a psychiatric context; (viii) that are not comprehen- sive, i.e. only focus on individual areas such as economy or acceptance or satisfaction or usability; (ix) which refer exclusively to the implementation and not the evaluation of the implementation.

Identifying relevant frameworks All systematic search results were imported into End- Note X8 and reimported into the Excel screening work- book by VonVille [15]. A multi-step screening process was performed. The first step included screening 100 ti- tles and 100 abstracts. The eligibility criteria were then refined. All titles were screened in the second step and the remaining abstracts in the third step. The eligibility criteria were then refined again before screening the full texts. If an identified article only applied a framework, for which the development is described in another art- icle, this was an intermediate step for the identification of the framework. The original article describing the framework was than identified and included in the fur- ther steps of the analysis. The narrative search was performed with the know-

ledge and eligibility criteria of the first screening process, which enabled a much more precise identification process. Google scholar was searched with the terms “framework” AND “evaluation” AND “nursing”. Articles were screened and reference lists were also examined, snowballing through the reference lists of these articles. Reference lists of the systematically identified studies and literature from a previous search [4] were also in- cluded. This method of snowballing is important for such complex search fields. It helped to obtain all rele- vant information on frameworks as a supplement to those not found in the systematic approach [16]. The expert consultation additionally focussed on the

identification of relevant frameworks. The experts were invited to name frameworks known to them and list all frameworks used in their projects to evaluate digital nursing technologies. The whole search and identifica- tion process can be found in Fig. 1.

Data extraction Purpose, perspective, and success definitions The first step for data extraction was to screen all full texts of the frameworks for the technology group to which they refer, the stated purpose and the evalu- ation perspective [17], as well as the success defin- ition/description (ii). This article defines the purpose of a framework as the description of what the frame- work is intended to achieve. The perspective describes the viewpoint from which the framework was devel- oped and thus the viewpoint from which the evalu- ation results could be interpreted. The analysis of “success” focused on the definitions or descriptions of what the articles mean by “success” or “successful technology”. A qualitative synthesis was conducted to identify and categorize the included purposes, per- spectives and success definitions by using textual nar- rative synthesis [18].

Table 1 Search terms of the systematic search in PubMed

Term Term Term Hits

Framework AND Evaluation AND Technol* 1381

Framework Evaluation ICT 24

Framework Evaluation Robot 48

Framework Evaluation Sensor 135

Framework Evaluation Telecare 10

Framework Evaluation Telemed 97

Framework Evaluation Digital 260

1955

Krick BMC Nursing (2021) 20:146 Page 3 of 19

Strengths and weaknesses A strengths and weaknesses analysis of the frameworks under consideration was also carried out to answer re- search question (iii). The criteria for evaluating the frameworks were developed in an expert workshop with experienced researcher in the field of evaluation. The criteria are listed in Table 2. This approach was chosen because there are no universal quality criteria for the strengths and weaknesses of DNT related evaluation frameworks in the literature.

Areas of evaluation To answer research question (iv) all frameworks were screened for similarities in their categorization systems. Most frameworks used different sorting systems and systematization logics. An iteratively developed data ex- traction form was drawn up in Excel and piloted with

three frameworks. It was decided to assign all evaluation topics of the frameworks to the generic categories “Top Category” “Subcategories” and “Specification” (Add- itional file 1). This step was necessary, because there is no universal systematization to categorize the content of evaluation frameworks, but there is a kind of hierarchy that can be found in these frameworks.

Charting the data All extracted information on the technology group to which the frameworks refer, the stated purpose and the evaluation perspective, as well as the definition/descrip- tion of success (research question ii), were charted in excel and listed with the respective framework. To answer research question (iv) all identified top cat-

egories were analyzed to build overarching top categories that were used for the systematization of these

Fig. 1 Search results and framework selection process

Krick BMC Nursing (2021) 20:146 Page 4 of 19

frameworks. The analysis of all top categories of the 18 frameworks resulted in 9 generic top categories, which were then defined and formed the basis for further ana- lysis. The definitions of these categories were generated inductively and iterative while analyzing all included evaluation aspects of the frameworks (analysis in Add- itional file 2; definitions in Table 3). Despite this process a non-overlapping categorization of these categories was not possible due to the complexity of the frameworks con- tent and the interconnectedness of different categories. In the next step, the extracted content of the subcat-

egories from the frameworks (Additional file 2) was ana- lyzed to identify similarities and differences related to the newly built top categories (Table 3). During the

extraction of the category “specifications”, these were found to be vastly diverse and, hence, not suitable for any standardization. Therefore, only the content of the subcategories was included in the mapping process. Where there were no subcategories, the top categories were included into this step. This was the case for the: Design and Evaluation of DHI Framework [14], Evalu- ation Framework for Fit-For-Purpose Connected Sensor Technologies [19], Digi-HTA [20], CISSM [21] (in parts because there were only specifications in the form of specific questions for the top categories). If the subcategories corresponded to a completely dif-

ferent sorting logic as the other frameworks and there- fore did not contain any evaluable information, the “specifications” were evaluated if they contained valid in- formation. This was done for the: Khoja–Durrani–Scott Evaluation Framework [22], the layered telemedicine im- plementation model [23]; and the Comprehensive evalu- ation framework for telemedicine implementation [24]. All charting results can be found in (Additional file 2).

The assignment may differ from their logic in the repre- sentation to the original assignment because the frame- works used different sorting systems and logics, which were unified in this representation. Models and frame- works often develop their own categorization logic. There is no uniformly recognized logic.

Results Search results The systematic search in PubMed generated 1957 hits. After removing the duplicates, 1.755 remained for screening the titles. The abstracts of 113 articles were chosen for screening, yielding 69 full-texts eligible for full-text screening. The systematic search generated a

Table 2 Guiding categories for assessing the strengths and weaknesses of the frameworks

Guiding Category

Content

Focus of the framework

Description of the specific focus of the framework. This can include a - description of the purpose (and the addressed question)

- the application setting - the technology (area)

Illustration - Clarity/ complexity of illustration - Visualization of connections and relationships within the framework

Terminology - Transparent definitions of terms and key concepts

Instructions for use

- Concrete application strategy and instructions for use

- Instruction on how the results can be interpreted

Scientific quality - Transparency of development process - Reflection of the limitations of the framework - Transferability of the framework (Settings, technologies, questions)

Table 3 Definitions of the DNT evaluation areas

Focus Product/Technology Objective Value/Effect Individual Organization

This area includes what the technology focuses on in terms of its objectives and purpose and the problems and needs it aims to solve for a specific target group in a specific setting.

This area includes all aspects of the technology itself. This ranges from visual appearance to functionality and certain specific technological aspects such as interoperability. (However, there is also an interface to the category “individual”, because certain individually perceived aspects are covered here, such as usability and access).

This category includes the relevant information on evidence, aspired values as well as intended and unintended effects of the technology.

This area includes reactions and perceived impressions, as well as the behaviour and the relationship of individuals towards the technology.

This area includes aspects that are relevant in the relationship between the technology and an organization.

Societal Ethics Economics Strategic

This area includes relevant aspects of the technology in a societal context (e.g. political, juridical, regulatory, or socio- cultural aspects - Overlaps with the area of ethics are possible).

This area includes relevant ethical standards and ethical implications to be considered in relation to the technology.

This area includes relevant economic aspects for the technology (e.g. business model, price, economic evaluation).

This area includes strategic aspects that may be relevant for the introduction and dissemination of the technology.

Krick BMC Nursing (2021) 20:146 Page 5 of 19

total of 22 articles to be analyzed for relevant frame- works. The narrative search and the expert consulta- tions yielded 13 articles with frameworks. There remained 26 frameworks for the final analysis process, after sorting out the duplicates generated by the dif- ferent search processes. Eight further frameworks were discarded during the data extraction process be- cause detailed analysis revealed that they did not meet the eligibility criteria. This left 18 frameworks filtered out for the final analysis (Fig. 1).

Analysis results Eighteen comprehensive evaluation frameworks that can be applied to DNT are presented in the results section. According to the technology categories there are n = 7 frameworks related to information and communication technologies (with different sub-sectors), n = 3 frame- works for telemedicine/telecare, and n = 1 framework for sensor technologies. The remaining frameworks were generalistic frameworks from the areas of digital health (n = 3), health (and care) technologies (n = 2), e-health (n = 1) and clinical informatics (n = 1). The classification was based on the technologies derived from the articles with reference to the definitions of technology categories from Krick et al. 2019 [2] (Table 3). The final selection of frameworks can be found in Table 4.

Purpose and perspectives of the frameworks The purposes and the perspectives of the selected frame- works were analyzed to answer research question (ii).

Although these frameworks have the common purpose of assessing digital technologies using specific assess- ment categories, the purposes described beyond this dif- fer. The detailed assessment can be found in Table 5. The qualitative synthesis identified 7 overarching cat- egories of purposes:

1. Help and guide researchers (design and evaluation process) [14, 22, 25–27, 32–35]

2. Identify success and failure factors (and help to manage them) [14, 19, 21, 23, 27, 34]

3. Assess the performance/success of a technology (outcomes, impact, errors, deficiencies) [28–31]

4. Make the results comparable [19] 5. Contribute to the quality and development of the

technology [31, 34, 35] 6. Support the implementation of a technology [33–

35] 7. Help in decision-making [20, 24, 31]

A further classification of the purposes could be made by dividing them into two main categories. (A) knowledge-oriented purposes (1–4) that mainly indi- cate that the frameworks and their use serve to gen- erate a certain form of knowledge. (B) practice-related purposes (5–7). Theses purposes could be summa- rized as application-oriented knowledge as they indi- cate that the knowledge will be used for a specific action such as development, implementation or deci- sion making.

Table 4 Included Frameworks and technology categories

Information and Communication technologies

Infoway benefits evaluation Framework [25]

Health Information Technology Evaluation Framework (HITREF) [26]

Hospital Information System Success Framework [27]

Development of an Evaluation Framework for Health Information Systems (DIPSA Framework) [28]

Human, Organization, Process and Technology-fit (HOPT-FIT) [29]

Clinical Information Systems Success Model (CISSM) [21]

Adapted nursing care performance framework [30]

Telemedicine/Telecare Model for Assessment of Telemedicine (MAST Manual) [31]

Comprehensive evaluation framework for telemedicine implementation [24]

The layered telemedicine implementation model [23]

Sensor Technologies Evaluation Framework for Fit-For-Purpose Connected Sensor Technologies [19]

Digital Health Design and Evaluation of DHI Framework [14]

Health technology assessment framework for digital healthcare services (Digi HTA) [20]

Digital Health Score Card [32]

Health (and care) technologies Health Technology Adoption Framework [33]

Nonadoption, Abandonment, Scale-up, Spread, and Sustainability Framework (NASSS Framework) [34]

E-health programs Khoja–Durrani–Scott Framework for e-Health Evaluation [22]

Clinical informatic interventions RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) (expanded to clinical informatics)) [35]

Krick BMC Nursing (2021) 20:146 Page 6 of 19

For a deeper understanding of the purpose of a frame- work, it is necessary to also analyze the perspective that the framework takes as it might have an influence on how the framework is intended to achieve that purpose. Of the 18 frameworks, n = 9 had a universal perspec-

tive, which means that they can be applied to different perspectives or remain neutral thus leaving the decision to the evaluator. The remaining nine frameworks indi- cate or define for themselves, that they were developed from a specific viewpoint, and that the results could be interpreted from this viewpoint. Of these nine, n = 4 take the perspective of a healthcare organization, n = 2 de- scribe the healthcare system as a perspective, n = 2 have the nurse’s perspective as a viewpoint and n = 1 is devel- oped from an investment program perspective.

Success definitions/descriptions The frameworks were also scrutinized for definitions or descriptions of what is meant by “success” or “successful technology” in order to gain a better understanding of the differences that might exist (Table 5). Most of the frameworks do not have an explicit definition of success [19, 20, 22, 26, 28–33]. In such cases, the evaluation cri- teria described in the frameworks and the resulting in- terpretation of the evaluation results could be used to make a statement about whether or not the technology in question was successful. The qualitative synthesis of the success definitions/ de-

scriptions of success identified three categories:

1. Success is when the technology achieves its intended purpose [27]

2. Success means achieving implementation, dissemination and/or sustainability of a technology [14, 23, 24, 34]

3. A successful technology must generate a net benefit [21]

Strengths and weaknesses of the frameworks The strengths and weaknesses of the frameworks under consideration were assessed in order to contribute to a better understanding of what constitutes a good (DNT) evaluation framework and answer research question (iii). Previously developed quality criteria for DNT Frame- works were used for the assessment (as described in the methods section). A detailed analysis of the assessment can be found in Table 6. (+) stands for strength and (−) denotes a weakness in a certain assessment area. The assessment revealed differences and similarities

between the frameworks under study. All frameworks included a description of their intended purpose and the question(s) addressed. Of the 18 frameworks, 14 do not explicitly describe an application setting, since these frameworks were developed with generic setting

approaches. They therefore received a (+) in the evalu- ation for the universality of the setting and at the same time a (−) because they are not specifically related to a setting. However, most frameworks describe specific technology areas on which they focus. Only six have uni- versal designs, for potentially different technologies, so that they have been assessed with a (+) for universality and a (−) for being non specific. Due to their elabor- ation, n = 12 of the frameworks are easily transferable to other contexts. N = 11 frameworks have a very clear vis- ual presentation and n = 9 include a visual representa- tion of connections and relationships of individual aspects within the framework. On the other hand, this means that n = 7 frameworks were not without visual weaknesses and n = 9 frameworks did not show visual connections between the aspects with which they were concerned. Almost all frameworks (n = 15) included transparent definitions of terms and key concepts and are transparent in terms of the development process (n = 14). However, many of the frameworks have weak- nesses in applicability, clear guidance, and assistance for the interpretation of the results. Only n = 9 frameworks are strong in the description of an application strategy and instructions for use and only n = 6 include soundad- vice on how to interpret the results. Furthermore, many articles on the frameworks do adequately discuss weak- nesses and limitations - if at all(n = 11). The frameworks with the highest scores across all assessment categories were:

(1.)Health Technology Adoption Framework [33] strengths n = 9, weaknesses n = 1 (no visualization of connections and relationships within the framework) and strength/weaknesses n = 1 (transferability limited to surgical context).

(2.)CISSM [21] strengths n = 9, weaknesses n = 1 (no description of the limitations) and strength/ weaknesses n = 1 (transferability limited to hospital context).

(3.)NASSS Framework [34] strengths n = 8, and strengths/weaknesses n = 3 (no clear focus on an application setting, no clear focus on a technology (area), and no applicable strategy using the framework.)

The detailed analysis of the assessment can be found in Table 6. More detailed descriptions of the strengths and weaknesses can be found in Additional file 3.

Areas of evaluation in relation to the assigned perspectives A detailed analysis of the areas of evaluation included in the frameworks was carried out to answer research

Krick BMC Nursing (2021) 20:146 Page 7 of 19

Table 5 Analysis and assessment of the frameworks

Analysis

Technology Field

Framework Authors/ Year

Perspective Stated Purpose Success Definition/ Description

Information and Communication technologies (Health Information Systems (HIS))

Infoway benefits evaluation Framework

Francis Lau et al. 2007 [25]

Investment programs for digital technologies (to guide evaluations)

1. Provide a high-level evi- dence based model to guide subsequent field evaluation

Success measured by analysing the results of the evaluation (Factors based on the van der Meijden et al. model [36])

Information and Communication technologies (Health Information Technologies (EHR))

Health Information Technology Evaluation Framework (HITREF)

Sockolow et al. 2012 [26]

Universal perspective (mainly influenced by health services research and informatics)

1. Conceptual tool for framing evaluations studies in assessing EHR-based imple- mentations in organizational, systematic, and environmental contexts 2. Displaying evaluation criteria

No success definition (measuring the success by analysing the results of the evaluation)

Information and Communication technologies (Hospital Information Systems)

Hospital Information System HIS Success Framework

Sadoughi et al. 2013 [27]

Universal perspective 1.Identification of Hospital Information System success and failure factors and the evaluation methods of these factors

Success as a dynamic concept. Success is when the technology achieves its intended purpose. (+ time, budget, and user satisfaction)

Information and Communication technologies (Integrated Health Information Systems (IHIS))

Development of an Evaluation Framework for Health Information Systems (DIPSA Framework)

Stylianides et al. 2018 [28]

Healthcare Organization

1.Evaluation framework for hospitals utilizing IHIS to help identify any existing deficiencies in the system

No success definition (measuring the success by analysing the results of the evaluation)

Information and Communication technologies (Health Information Systems)

Human, Organization, Process and Technology-fit (HOPT- FIT)

Yusof 2019 [29]

Healthcare Organization (focus on technology induced errors)

1. Evaluate HIS performance and efficiency 2. Systematically guide error evaluation 3. Describing the Human- Organization-Process-Technol- ogy fit

No success definition (measuring success with the included dimensions of HIS success)

Information and Communication technologies (Clinical Information Systems (CIS))

Clinical Information Systems Success Model (CISSM)

Garcia- Smith & Effken 2013 [21]

Nurse’s perspective 1. Framework for evaluating CIS success from the nurse’s perspective

Success = net benefit (“degree to which a nurse believes that using a particular system enhances job performance”)

Information and Communication technologies (Information and Communication technologies for nurses)

Adapted nursing care performance framework

Rouleau et al. 2017 [30]

Nurse’s perspective 1. Illustrate how ICTs interventions influence nursing care and impact health outcomes

No success definition (measuring the success by analysing the results of the evaluation)

Telemedicine/ Telecare

Model for Assessment of Telemedicine (MAST Manual)

Kidhom et al. 2010 [31]

Universal perspective (user-based decision making, research)

1. Describe effectiveness 2. Contribution to quality of care of telemedicine applications 3. Produce a basis for decision making

No success definition (measuring the success by analysing the results of the evaluation)

Telemedicine/ Telecare

Comprehensive evaluation framework for telemedicine implementation

Chang 2015 [24]

Universal perspective (decision making for individuals, organizations, and communities)

1. Summarising important themes for the evaluation of telemedicine systems 2. Support related stakeholders’ decision-making by promoting general under- standing, and resolving argu- ments and controversies

Long-term implementation

Telemedicine/ The layered Broens et al. Universal perspective 1. Detailed classification of the Successful implementation

Krick BMC Nursing (2021) 20:146 Page 8 of 19

Table 5 Analysis and assessment of the frameworks (Continued)

Analysis

Technology Field

Framework Authors/ Year

Perspective Stated Purpose Success Definition/ Description

Telecare telemedicine implementation model

2007 [23] (the focus on individual determinants/ perspectives changes throughout the development life cycle)

determinants of the success of future telemedicine implementations

(“putting an idea or a concept into actual practice”)

Sensor Technologies (Connected Sensor Technologies: including wearables. biosensors)

Evaluation Framework for Fit-For-Purpose Connected Sensor Technologies

Coravos et al. 2020 [19]

Healthcare System Perspective (users and other stakeholders)

1. Working evaluation framework that reflects different types of risks 2. Framework is conducted to better manage these risks 3. Make information on sensor technologies more comparable and understandable

No success definition (success could be measured by analysing the results of the evaluation and comparing them with the standards for connected sensors)

Digital Health (Digital Health Interventions (DHI))

Design and Evaluationof DHI Framework

Kowatsch et al. 2019 [14]

Universal perspective (researchers and practitioners)

1. Framework for the design and evaluation of DHI 2. Showing evaluation criteria and implementation barriers to be considered during the life cycle phases of DHI 3. Support researchers and practitioners from conception to large-scale implementations

A successful DHI needs to consider “the selection of suitable evaluation criteria and the overcoming of implementation barriers”

Digital Health (Digital Healthcare Services: mHealth, AI, and robotics)

Health technology assessment framework for digital healthcare services (Digi HTA)

Jari et al. 2019 [20]

Healthcare System Perspective (decision making)

1. Inform decisionmakers in order to better support the introduction of new health technologies

No success definition (success could be measured by analysing the results of the evaluation)

Digital Health (Digital Health Technologies)

Digital Health Score Card

Mathews et al. 2019 [32]

Universal perspective (multi-stakeholder approach)

1. Multi-stakeholder approach to objectively evaluate digital health solutions

Measuring the success by analysing the results of the evaluation (Success as the successful delivery of validated digital health solutions)

Health (and care) technologies

Health Technology Adoption Framework

Poulin et al. 2013 [33]

Healthcare Organization

1. Framework with clear, user- validated criteria for evaluating new health technologies for adoption at the local level

No success definition (success could be measured by analysing the results of the evaluation)

Health (and care) technologies

Nonadoption, Abandonment, Scale- up, Spread, and Sus- tainability Framework (NASSS Framework) [34]

Greenhalgh et al. 2017

Universal perspective 1. Framework to help predict and evaluate the success of a technology-supported health or social care program 2. Help to design, develop, implement, scale up, spread, and sustain technology- supported health or social care programs by identifying key challenges in different do- mains and the interactions be- tween them

Adoption, scale-up, spread, and sustainability of a technology

E-health programs

Khoja–Durrani–Scott Framework for e- Health Evaluation

Khoja et al. 2013 [22]

Universal perspective (included tools usable for managers, healthcare providers, and clients)

1. Comprehensive Framework to show relevant themes for e- health evaluation

No success definition (measuring the success by analysing the results of the evaluation)

Clinical informatic interventions

RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) (expanded to clinical informatics)

Bakken & Ruland 2009 [35]

Healthcare Organization (implementation in organizational practice)

1. Used to design, implementation, evaluation, and reporting of clinical informatics with a goal of translation of research into practice

No success definition (measuring the success by analysing the results of the evaluation)

Krick BMC Nursing (2021) 20:146 Page 9 of 19

Ta b le

6 St re ng

th s an d W ea kn es se s

+ = st re ng

th -=

w ea

kn es s

+ /−

= st re ng

th an

d w ea

kn es s

C ri te ri a

Su m m ar y

Fr am

ew or ks

D es cr ip tio

n of

th e

pu rp os e (a nd

th e

ad dr es se d

qu es tio

n( s)

D es cr ip tio

n of

th e

ap pl ic at io n

se tt in g

D es cr ip tio

n of

th e

te ch no

lo gy

(a re a)

C la rit y/

co m pl ex ity

of ill us tr at io n

Vi su al iz at io n of

co nn

ec tio

ns an d

re la tio

ns hi ps

w ith

in th e

fra m ew

or k

Tr an sp ar en

t de

fin iti on

s of

te rm

s an d ke y

co nc ep

ts

C on

cr et e

ap pl ic at io n

st ra te gy

an d

in st ru ct io ns

fo r us e

In st ru ct io n

on ho

w th e re su lts

ca n be

in te rp re te d

Tr an sp ar en

cy of de

ve lo pm

en t

pr oc es s

Re fle ct io n

of th e

lim ita tio

ns of

th e

fra m ew

or k

Tr an sf er ab ili ty

of th e

fra m ew

or k

(S et tin

gs ,

te ch no

lo gi es ,

qu es tio

ns )

+ –

+ /−

H ea lth

Te ch no

lo gy

A do

pt io n Fr am

ew or k [3 3]

+ +

+ +

– +

+ +

+ +

+ /−

9 1

1

C lin ic al In fo rm

at io n

Sy st em

s Su cc es s M od

el (C IS SM

)[ 21 ]

+ +

+ +

+ +

+ +

+ –

+ /−

9 1

1

N on

ad op

tio n,

A ba nd

on m en

t, Sc al e- up

, Sp re ad ,a nd

Su st ai na bi lit y

Fr am

ew or k (N A SS S

Fr am

ew or k) [3 4]

+ + /−

+ /−

+ +

+ + /−

+ +

+ +

8 0

3

H ea lth

In fo rm

at io n

Te ch no

lo gy

Ev al ua tio

n Fr am

ew or k (H IT RE F)

[2 6]

+ + /−

+ +

+ +

+ –

+ –

+ 8

2 1

Ev al ua tio

n Fr am

ew or k fo r

Fi t- Fo r- Pu

rp os e C on

- ne

ct ed

Se ns or

Te ch no

lo -

gi es

[1 9]

+ + /−

+ +

– +

+ +

+ –

+ 8

2 1

H os pi ta lI nf or m at io n

Sy st em

Su cc es s

Fr am

ew or k [2 7]

+ +

+ –

– +

+ –

+ +

+ 8

3 0

A da pt ed

nu rs in g ca re

pe rfo

rm an ce

fra m ew

or k

[3 0]

+ + /−

+ +

+ +

– + /−

– +

+ 7

2 2

Th e la ye re d te le m ed

ic in e

im pl em

en ta tio

n m od

el [2 3]

+ + /−

+ +

+ +

+ /−

– +

+ –

7 2

2

M od

el fo r A ss es sm

en t of

Te le m ed

ic in e (M

A ST

M an ua l) [3 1]

+ + /−

+ –

– +

+ –

+ +

+ 7

3 1

H ea lth

te ch no

lo gy

as se ss m en

t fra m ew

or k fo r

di gi ta lh

ea lth

ca re

se rv ic es

(D ig iH

TA ) [2 0]

+ + /−

+ –

– +

+ +

+ –

+ 7

3 1

In fo w ay

be ne

fit s

ev al ua tio

n Fr am

ew or k

[2 5]

+ + /−

+ /−

+ +

+ –

– +

– +

6 3

2

Krick BMC Nursing (2021) 20:146 Page 10 of 19

Ta b le

6 St re ng

th s an d W ea kn es se s (C on

tin ue d)

+ = st re ng

th -=

w ea

kn es s

+ /−

= st re ng

th an

d w ea

kn es s

C ri te ri a

Su m m ar y

D es ig n an d Ev al ua tio

n of

D H IF ra m ew

or k [1 4]

+ + /−

+ /−

– –

+ +

– +

+ +

6 3

2

RE -A IM

(R ea ch ,

Ef fe ct iv en

es s, A do

pt io n,

Im pl em

en ta tio

n, an d

M ai nt en

an ce ) (e xp an de

d to

cl in ic al in fo rm

at ic s) )

[3 5]

+ + /−

+ /−

– –

+ +

+ +

– +

6 3

2

D ev el op

m en

t of

an Ev al ua tio

n Fr am

ew or k fo r

H ea lth

In fo rm

at io n

Sy st em

s (D IP SA

Fr am

ew or k) [2 8]

+ +

+ –

– +

– –

+ –

+ 6

5 0

Kh oj a– D ur ra ni –S co tt

Fr am

ew or k fo r e- H ea lth

Ev al ua tio

n [2 2]

+ + /−

+ /−

– +

+ –

– +

– +

5 4

2

D ig ita lH

ea lth

Sc or e C ar d

[3 2]

+ + /−

+ /−

+ +

+ –

– –

– –

4 5

2

H um

an ,O

rg an iz at io n,

Pr oc es s an d Te ch no

lo gy -

fit (H O PT -F IT ) [2 9]

+ + /−

+ +

+ –

– –

– –

– 4

6 1

C om

pr eh

en si ve

ev al ua tio

n fra m ew

or k fo r

te le m ed

ic in e

im pl em

en ta tio

n [2 4]

+ + /−

+ +

– –

– –

– –

– 3

7 1

Krick BMC Nursing (2021) 20:146 Page 11 of 19

question (iv). The analysis resulted in definitions for nine evaluation areas that are described in Table 3. Table 7 shows a comparison of the frameworks

regarding the evaluation areas they cover. The re- sults of the analysis of these areas indicate where the frameworks have their main areas of focus. The colour coding in the table signals that a framework covers a certain area. The allocation was based on the definitions and the sorting logic described in the Methods section above. The specific perspective described for the frameworks was also included in the table to crosscheck whether it is possible to make generalized statements about the existence of

certain evaluation areas in relation to the perspec- tive taken. Most frameworks (n = 17) contained evaluation aspects

of the area of objective value/effect. Also, the evaluation of the specific product/technology aspects (n = 16), as- pects of the organization (n = 15) and the relationship of individuals to the technology (n = 14) was largely represented. Societal (n = 10) and strategic (n = 6) aspects, as well as

ethical aspects (n = 4), were not as frequent. A closer look reveals that these aspects are particularly rare when the perspective described is the healthcare organization or the nurses. At the same time, these aspects are jointly

Table 7 Frameworks with evaluation areas and perspectives

Krick BMC Nursing (2021) 20:146 Page 12 of 19

represented three times in the assessment if the perspec- tive “universal” was assigned (in the HIS Success Frame- work [27], MAST Manual [31] and the Khoja–Durrani– Scott Framework for e-Health Evaluation [22]). Overall the evaluation area “focus” is not as frequently

represented. Only n = 5 frameworks contain aspects of this areas. All frameworks containing this aspect come from the “universal” perspective. The area covers evalu- ation aspects that can be used as starting point for the design of a DNT or DNT evaluation by conducting a “needs analysis” (related to the addressed problems and needs of a target group). Frameworks for which a universal perspective has been

described (n = 9) also cover more areas more often in general (coverage of 7,2 areas on average). While frame- works for which a healthcare organization perspective is described (n = 4) only cover 4 areas on average (none of them covers the areas focus, societal or ethics), and frameworks covering the nurse’s perspective (n = 2) only cover 2 and 4 areas respectively (none of them covers the focus, societal, ethics, economic or strategic aspects). The frameworks with a healthcare system perspective (n = 2) cover 4 areas on average (none of them covers the focus, individual, organization, ethics, or strategic areas) and the only framework with an investment pro- gram perspective covers 5 areas (does not include the areas focus, societal, ethics or strategic). A mapping of the content of the frameworks to the evaluation areas was carried out and can be found in Table 6.

Discussion The aim of this study was to contribute to a better un- derstanding of what constitutes a good (DNT) evaluation framework and to guide field researchers in the selection and application of evaluation frameworks. This aim re- sulted in four research questions: (i) Which comprehen- sive evaluation frameworks that can be applied to DNT evaluation are available in the literature? (ii) What pur- poses, perspectives, and definitions of success are de- scribed in these frameworks? (iii) What are the strengths and weaknesses of the included frameworks? (iv) Which areas of evaluation are represented in the frameworks and where are the most overlaps and differences be- tween these frameworks? Eighteen different comprehensive evaluation frame-

works were identified that met the inclusion criteria in the field of DNT. Unlike other overviews of evaluation frame- works, which either had a very technology-specific focus (e.g. on health information systems [37]) or a different thematic orientation (e.g. on HTA [38]), this article took a broad approach on comprehensive frameworks for DNT. This led to the identification of technology-specific

frameworks n = 11 (Information and Communication technologies, telemedicine/telecare, and sensor

technologies) – as well as more generalistic frame- works addressing digital health (n = 3), health (and care) technologies (n = 2), e-health (n = 1) or clinical informatics (n = 1). The identified frameworks and their main features are listed in the Tables 4–7. These may serve as a good overview and starting point for researchers to select an appropriate framework. It should be noted, however, that although a specific

definition of DNT was used, there is inevitably an over- lap to other different themes like e-health or digital health which makes a general distinction very difficult. The frameworks for specific technology categories in this article only cover a part of the technologies that can be subsumed under DNT. For example, evaluation frame- works for monitoring technologies, assistive devices or ambient assisted living, are not included [2]. This is so as to avoid too much heterogeneity of technologies in this article and the technologies included make up a sig- nificant proportion of the DNTs discussed in the litera- ture [2]. Also, the generic frameworks can potentially be used for the evaluation of further technologies. To distinguish, differentiate and select relevant frame-

works, researchers should look at specific assessment categories and the frameworks content. Some important aspects regarding the frameworks are covered in this art- icle. The perspective of a framework is essential to put the definitions of success in a framework into context and to understand a potential interpretation of the evaluation categories. There should also be clarity about the purpose of the framework to apply it appropriately, and transparency about where frameworks have their re- spective strengths and weaknesses. All these issues were analyzed and are discussed in the following.

Purposes It is important to understand that although the common purpose of the frameworks dealt with here is to assess digital technologies using specific evaluation categories, the purposes described beyond this differ. Most frame- works included here were developed (i) to help and guide researchers in the design and realisation of an evaluation [14, 22, 25–27, 32–35], and/or (ii) to support the identification of success and failure factors (and to help manage them) [14, 19, 21, 23, 27, 34]. (iii) Four frameworks were specifically designed to assess the per- formance/success of a technology (outcomes, impact, er- rors, deficiencies) [28–31], but only one framework (iv) was drawn up to help to make the results comparable [19]. This could be since the other articles simply as- sume the comparability issue to be implicitly logical and therefore do not name it explicitly as a purpose. These four purpose categories (with the exception of the man- agement of success and failure factors) could be

Krick BMC Nursing (2021) 20:146 Page 13 of 19

summarized as knowledge-oriented purposes, indicating that the frameworks mainly serve to generate a certain form of knowledge. Some frameworks indicate a more practice-related

purpose such as (v) contributing to the quality and de- velopment of the technology [31, 34, 35], (vi) supporting the implementation of a technology [33–35] or (vii) pro- viding help in decision-making [20, 24, 31]. These pur- poses can be summarized as application-oriented knowledge generation. As the classification shows, a DNT framework can have several purposes in both cat- egories: knowledge generation and application orienta- tion. The knowledge-oriented categories i, ii, iii, iv and the practice-related category vi are in line with seven out of eight general attributes of evaluation frameworks in healthcare identified by Bradford et al. (2019) These are 1. simplify a complex (evaluation) process, 2. provide structure (for an evaluation), 3. facilitate the evaluation process, 4. promote meaningful evaluation, 5. identify and explain outcomes, 6. generate transferable lessons, 7. identify mechanisms driving or inhibiting change [39]. Bradford and colleagues also consider it an important element that frameworks help to identify relevant stake- holders [39]. This element was not mentioned as a pur- pose by any of the frameworks in the present study. In general, DNT evaluation and DNT relevant evaluation frameworks can consider knowledge-oriented and application-oriented purposes. It does not make a frame- work better or worse if it includes only one of the cat- egories, but having a clearly defined purpose is a quality criterion when choosing a reliable framework. Re- searchers must be aware of the intended purpose of a framework when choosing their evaluation approach.

Perspective and evaluation areas A closer look at the perspectives of the frameworks dis- cussed here raises the question whether the perspective under which a framework has been developed, may have an impact on the evaluation categories included. The analysis of the frameworks revealed five perspectives: universal, healthcare system, healthcare organization, nurses, and investment program perspective. Of the 18 frameworks, n = 9 had a universal perspec-

tive, which means that they can be applied to different perspectives, leavingthe decision is left to the evaluator. Frameworks for which a universal perspective has been described cover more evaluation areas in general (cover- age of 7,2 areas on average), what supports this assump- tion. Frameworks for which a healthcare organization perspective is described only cover 4 areas on average (none of them covers the areas focus, societal or ethics), and frameworks with the nurse’s perspective only cover 3 areas on average (none of them covers the aspects focus, societal, ethics, economic or strategic). Those

frameworks with a healthcare system perspective (n = 2) cover 4 areas on average (with none of them covering the areas focus, individual, organization, ethics, or stra- tegic) and the only framework with an investment pro- gram perspective covers 5 areas (does not include the areas focus, societal, ethics or strategic). The perspective with which a framework has been developed is therefore always an important feature to consider when selecting a framework, as this could influence the evaluation aspects included. In addition, there are several stakeholder perspectives

that were not taken as the main perspective in the frameworks. These perspectives are the payors perspec- tive, the perspective of the patient / person in need of care and the perspective of the informal caregiver. All these perspectives are particularly important in the con- text of DNTs. However, it should be mentioned that these perspectives are often included in the universal frameworks like the patient perspective in the MAST [31] or the 3rd party payment aspect in the comprehen- sive evaluation framework for telemedicine implementa- tion [24]. In summary, when selecting and using a comprehensive evaluation framework to evaluate a DNT, the perspective of the chosen framework and the intended perspective of the evaluation should always be reflected and contrasted, as there may be a relationship between the perspective and the evaluation areas in- cluded in the framework. The frameworks with a narrow perspective, such as the nurses’ perspective, cover fewer evaluation areas overall in this study, while those with a universal perspective cover significantly more. Three ex- amples explaining this in more detail can be found in the discussion section entitled “Discussion of the three most relevant frameworks”. If researchers require a com- prehensive framework with as many evaluation areas as possible, they must choose a framework with a universal evaluation perspective.

Success definitions/descriptions It is not easy to define a successful digital nursing tech- nology. Nguyen et al. (2014) argue that the success of a technology may be “disputed depending on the interests of the evaluating party.” [40] This is in line with the suc- cess description of Lau (2009) who sees it in the context of an “ongoing negotiation and adaptation of interrela- tionships” of the healthcare professionals involved [41]. A large multi stakeholder Delphi study conducted by McNair and colleagues (2006) [42] concluded that “suc- cess cannot be characterized along one single axis” and therefore defined success as the fulfilment or non- fulfilment of five consensus based aspects, namely (1) the wide usage in daily practice, (2) the fulfilment of the role and tasks it was planned for (in a specific environ- ment), (3) the support of good medical practice

Krick BMC Nursing (2021) 20:146 Page 14 of 19

(benefitting the patient), (4) the benefits to the health- care organization and working conditions, (5) easy up- gradability to adapt to the developments in practice [42]. Aspects 1–4 from McNair and colleagues were also

identified as success definitions in the analyzed frame- works in this article (1. Success is when the technology achieves its intended purpose [27]; 2. Success means achieving implementation, dissemination and/or sustain- ability of a technology [14, 23, 24, 34]; 3. A successful technology must generate a net benefit [21]). This sug- gests that these aspects could be universal definitions of success for digital nursing technologies. Aspect 5 of the McNair study could be added as equally important. However, looking at success from a certain perspective

in a specific context might lead to additional perceptions or definitions of success – which could influence the evaluation. In view of these differences in definition, it must be

concludedthat the “success” of a DNT is a relative term, made up of various aspects and depths of success defini- tions, the selection of which depends strongly on the evaluation perspective. This insight is decisive for the evaluation of a DNT because it should always be reflected for whom and from which perspective an evaluation is conducted and what is considered as successful.

Strengths and weaknesses of the frameworks The strengths and weaknesses analysis process in this article identified framework components where more guidance would be beneficial, and which are important to consider when selecting a DNT evaluation framework. The assessment included the focus of a framework, the illustration, terminology, instructions for use and scien- tific quality. The strengths and weaknesses criteria were created especially for DNT evaluation frameworks al- though there are other quality criteria that could be ap- plied from other healthcare fields. For example, Bradford et al. (2019) [39] provide 6 quality criteria for frame- works, most of which are similar to the criteria in this article. Bradford et al. also suggest assessing whether the frameworks help to identify and include stakeholders as well as mechanisms that drive or inhibit change – which might be also helpful selecting a framework. The results of the framework assessment presented in

this article need to be seen in a wider context. The evaluation was carried out with a view to assessing of perceived strengths and weaknesses in general. Several aspects mighthave been rated differently in other con- texts, e.g., assessing the transferability of a framework to only onespecific context, as with the Health Technology Adoption Framework [33] and the CISSM [21] as a strength or a weakness depends on the viewpoint of the planned evaluation. If the context is surgical the Health

Technology Adoption Framework might be the perfect choice, although it is not transferable to other contexts. By showing and applying assessment possibilities for

DNT frameworks this article gives guidance for the se- lection of appropriate DNT evaluation frameworks. An assessment of strengths and weaknesses in combination with the other important information presented in this article is crucial.

Discussion of the three most relevant frameworks This section describes the three highest rated frame- works of the strengths and weakness analysis in more detail in order to give researchers insights into the ana- lytical dimensions described in this article. The complete data for all frameworks are summarized in Tables 4–7. The description is written in an application-oriented way to support researchers. Problematic aspects in the selection of DNT frameworks are also discussed.

Framework 1: health technology adoption framework [33] The Health Technology Adoption Framework shows val- idated criteria for assessing new health technologies for adoption at local level with a hospital focus in the surgi- cal context. It is developed generically with respect to the technology to be assessed and can therefore be ap- plied to several DNTs. The evaluation categories presented focus on the

product/technology, its objective value/effect in the adoption process, the evaluation of organizational as- pects as well as economic and strategic aspects with re- spect to the (potential) adoption process. Individual, societal, ethical and aspects of the area “focus” are not covered. This distribution of the aspects considered could be related to the fact that the framework was de- veloped from the perspective of the healthcare organization and is intended to help decision makers (e.g., nurse managers) to evaluate the suitability of new technologies as well as to facilitate smooth adoption from the perspective of the organization. This means that while aspects such as security, costs and strategic fit in the organization are considered, individual aspects like usability or acceptance are not. The Health Technol- ogy Adoption Framework covers more evaluation areas (n = 5) than the other three frameworks in this study with a health organization perspective (see Table 7). Overall, however, all frameworks with this perspective cover fewer areas than those with a universal perspec- tive. This limitation regarding the evaluation areas should always be considered, when deciding on a framework. The Health Technology Adoption Framework per-

forms very well in the framework quality assessment. The descriptions and definitions are accurate and complete. An evaluation tool with appropriate evaluation

Krick BMC Nursing (2021) 20:146 Page 15 of 19

categories, an application strategy, relevant questions, and a scoring logic is provided. The development process of the framework is also presented transparently, and the corresponding limitations are named in the art- icle. Only the transferability to other settings is limited and there is no visual presentation of the framework. It is displayed in a table and not as a graphic. Overall, the Health Technology Adoption Framework provides a very good evaluation basis with valid assessment criteria.

Framework 2: clinical information systems success model (CISSM) [21] The CISSM framework is designed to assess the success of clinical information systems (CIS) from the nurses’ perspective. The framework is suitable for all DNTs from the CIS category in the hospital setting. CISSM fo- cusses only on a specific selection of evaluation areas: product/technology, objective value/ effect, individual as- pects, and aspects of the healthcare organization. The societal, ethical, economic and strategic aspects are not considered. Compared to the second framework with the nurses’

perspective from this study (Adapted nursing care per- formance framework) [30], the CISSM covers two more evaluation areas. The Adapted nursing care performance framework [30] only covers the areas of the objective value/ effect and individual aspects of technology. By comparison, CISSM is the more comprehensive frame- work - but overall, both cover only a few of the possible evaluation areas. It is certainly critical that none of the two frameworks covers ethical aspects. This should be considered when selecting and applying them, as the ethical aspects are also important in this context. Add- itional ethical evaluation criteria from other frameworks could be added to fill this gap. When selecting a framework with such a narrow per-

spective it should be borne in mind that aspects of the “bigger picture” might not be depicted. The specific suc- cess definition of a technology in this framework fo- cusses solely on the nurses’ net benefit. Success in this case is the “degree to which a nurse believes that using a particular system enhances job performance”. This makes the CISSM framework particularly suitable for use when the nurses’ perspective is the object of re- search. If a broader spectrum of evaluation aspects is to be researched, then this framework would not be the right choice. At the same time, however, the framework scores

very well in the strengths and weaknesses assessment for this specific perspective. It has a very clear de- scription of purpose, target setting, and technology addressed. It is very well illustrated, and the connec- tions are visualized. The key concepts and terms a are completely defined. A procedure and evaluation

matrix are provided as well as examples for the inter- pretation of an evaluation. The development process was fully explained. The main shortcoming of the article on this frame-

work is that no explanation is given about its limitations. Overall, the CISSM framework is a good basis to assess the success of clinical information systems (CIS) from the nurses’ perspective and the article provides a good basis and example for the methodology.

Framework 3: nonadoption, abandonment, scale-up, spread, and sustainability framework (NASSS framework) [34] The Nonadoption, Abandonment, Scale-up, Spread, and Sustainability Framework (NASSS) was designed to help predict and evaluate the success of health and care tech- nologies and to support the design, development, imple- mentation, scale up, spread, and sustainability of technology-supported health or social care programs by identifying key challenges in different domains and the interactions between them. As the description shows, this is the most comprehensive and generic framework of the three described. The framework does not focus on any specific technology or setting which makes it suit- able for any DNT. The comprehensiveness of the frame- work is also reflected in the included evaluation aspects. The framework considers aspects of the focus, product/ technology, objective value/ effect, individual aspects, as- pects of the organization, societal aspects, and economic aspects – which makes it a compendium of evaluation options for DNTs. Only the ethics and strategic aspects as defined in this review are not covered by the NASSS Framework. Thus, among all the frameworks covered in this study, the NASSS framework is one that covers the most evaluation areas. A special feature of this framework is that it addresses

the issue of complexity. It categorizes the interpretation of the evaluation domains as simple, complicated, or complex. The level of complexity of each evaluation as- pect is seen as the greatest challenge to scale-up, spread, and sustainability. No specific evaluation perspective is given, which makes it a universally applicable frame- work. However, this universality is also accompanied by a limitation. As no specific setting, technology or per- spective is given, the researcher using it must always re- flect on the individual application strategy. The NASSS Framework is not a directly applicable or formulaic in- strument, which is reflected by the authors of the frame- work themselves. Specific additional tools to solve this problem have been published recently [43]. Apart from this, the NASSS framework scores very

well in the strengths and weaknesses assessment. Looking at all three frameworks in comparison, the

question arises whether a framework with a specific

Krick BMC Nursing (2021) 20:146 Page 16 of 19

perspective is always preferable to a universal frame- work. This question can only be answered on a case-by- case basis when assessing the fit of a framework to the research situation.

Limitations of the article It is important to discuss the limitations of such an ex- tensive procedure as described in this article. A three- component search process was chosen for this article., namely (1) a Systematic search in PubMed, (2) a narra- tive search in Google Scholar and reference lists, (3) ex- pert consultations. Additional databases could have been systematically searched. Experience from previous sys- tematic search processes in such complex fields has shown that a combination of systematic searches and other methods is a good way to identify relevant articles [4]. However, with such a search procedure there is al- ways a chance that something is left undiscovered. There is also a limitation associated with a single researcher conducting a study that may have affected the search and analysis process. Single studies with frameworks might have been overlooked [44] or specific biases in in- terpretation could have occurred in the analysis process [12]. These limitations were sought to be minimised through expert workshops and consultations to identify relevant frameworks and discuss the methodology and the analysis process of this article. Also, the exclusion criteria were very strict. Only literature published in the English language was included. The DNT related criteria were based on the knowledge of Krick et al. 2019 [2] and Krick et al. 2020 [4] but still led to a heterogeneity of frameworks due to the broadness of the field. Only comprehensive, technology related frameworks

were included, which ruled out many specific frame- works that could potentially also have been included. Frameworks focussing exclusively on special fields like health economics or acceptance could have been in- cluded, because they are certainly relevant for DNT, but were excluded due to their specificity and the large number of different frameworks available. Generalistic evaluation frameworks for (complex) healthcare inter- ventions like the Consolidated Framework for Imple- mentation Research (CFIR) [45] or on HTA Frameworks could have also been included (e.g. Integrate HTA [46]) – but it was necessary to limit the included frameworks to allow statements on this specific field of research. The framework analysis itself has further limitations.

Due to the heterogeneity of the frameworks and the dif- ference in their content the categories and the assign- ment of the framework content to these categories is subjective. Even though the process of evaluation and classification was carried out with the utmost care, a non-overlapping categorization of these categories was not possible due to the complexity of the frameworks

content and the interconnectedness of different categor- ies. There are overlaps between the category of technol- ogy and individual as well as ethical and societal aspects. It should also be considered that the generalized state- ments made in this article can only provide initial indi- cations in this specific field. No statistical analysis was carried out, but a qualitative and hypothesis-generating analysis. Despite these limitations, this article makes an important contribution to further research in the field of DNT evaluation.

Conclusion This research article provides orientation in the complex field of DNT evaluation. Eighteen relevant comprehen- sive evaluation frameworks for DNT have been identi- fied. These frameworks focussed on different purposes and included various evaluation perspectives. The ana- lysis and mapping in this article provide a good overview of the frameworks under consideration, their similarities and differences, evaluation areas, success definitions, strengths, and weaknesses. The assessment whether a DNT evaluation framework is good should be based on the clarity of the description of these aspects and the ful- filment of the quality criteria described. The information on DNT evaluation frameworks pro-

vided in this review can therefore help in communication between decision makers and researchers to improve the evaluation process [11], by providing systematic informa- tion and a structure for the evaluation of a DNT. This can facilitate system implementation or provide helpful infor- mation in the technology development process. Compre- hensive evaluation has the potential to avoid early system failures, prevent wrong investment decisions [10] or con- tribute to the development and implementation of better, more useful DNTs from a societal perspective. The gen- eral question what evaluation frameworks can contribute to this process should be further analyzed. The diversity and heterogeneity of frameworks presented in this article shows, that there is not one sole definition of the term “evaluation framework” in the field of DNT. The question of what constitutes a good DNT framework could also be further explored taking the criteria developed in this art- icle as a starting point. Future research could also address questions regarding

what makes a successful DNT. The definition of a “suc- cessful” DNT, the role of the evaluation perspective and the purpose of the evaluation should be discussed when analyzing this question. Overall, evaluators can use the concise information

and quality criteria of this article as a starting point to select DNT evaluation frameworks for their research projects or to assess the quality of an evaluation frame- work for DNT, as well as a basis for exploring the open research questions raised.

Krick BMC Nursing (2021) 20:146 Page 17 of 19

Abbreviations CFIR: Consolidated Framework for Implementation Research; CIS: Clinical Information Systems; CISSM: Clinical Information Systems Success Model; DHI: Digital Health Interventions; DIPSA: Development of an Evaluation Framework for Health Information Systems; DNT: Digital nursing technologies; EHR: Electronic Health Records; HIS: Health Information System; HTA: Health Technology Assessment; HITREF: Health Information Technology Evaluation Framework; HOPT-FIT: Human, Organization, Process and Technology-fit; IS: Information System; ICT: Information and Communication Technology; IHIS: Integrated Health Information Systems; MAST: Model for Assessment of Telemedicine; mHealth: Mobile health; NASSS: Nonadoption, abandonment, scale-up, spread, and sustainability; NCPF: Nursing care performance framework; RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance

Supplementary Information The online version contains supplementary material available at https://doi. org/10.1186/s12912-021-00654-8.

Additional file 1. Framework Analysis.

Additional file 2. Evaluation Areas.

Additional file 3. Detailed strength and weakness analysis.

Acknowledgements This work is part of a project funded and supported by the Federal Ministry of Education and Research (BMBF). This research was supported by feedback of Kai Huter. The DNT experts consulted for this article were Anika Heimann-Steinert, Jürgen Zerth, Jörn Krückeberg, Ronny Klawunn.

Author’s contributions TK conceptualized and conducted the study. The author authorized the final version of the manuscript that was submitted. The author read and approved the final manuscript.

Funding This study arises from the project “Pflegeinnovationszentrum” (PIZ) which is funded by the Federal Ministry of Education and Research (BMBF) (Grant number: 16SV7821). Open Access funding enabled and organized by Projekt DEAL.

Availability of data and materials The datasets used and/or analyzed during the current study are included in this published article and the Additional files.

Declarations

Ethics approval and consent to participate Not applicable.

Consent for publication Not applicable.

Competing interests There are no competing interests to declare.

Received: 26 January 2021 Accepted: 16 July 2021

References 1. Huter K, Krick T, Domhoff D, Seibert K, Wolf-Ostermann K, Rothgang H.

Effectiveness of digital technologies to support nursing care: results of a scoping review. J Multidiscip Healthc. 2020;13:1905–26. https://doi.org/1 0.2147/JMDH.S286193.

2. Krick T, Huter K, Domhoff D, Schmidt A, Rothgang H, Wolf-Ostermann K. Digital technology and nursing care: a scoping review on acceptance, effectiveness and efficiency studies of informal and formal care technologies. BMC Health Serv Res. 2019;19(400).

3. Isfort M, Rottländer R, Weidner F, Tucman D, Gehlen D, Hylla J. Pflege- thermometer. Eine bundesweite Befragung von Leitungskräften zur situation der Pflege und Patientenversorgung in der ambulanten Pflege. Köln: Deutsches Institut für angewandte Pflegeforschung e.V. (dip); 2016.

4. Krick T, Huter K, Seibert K, Domhoff D, Wolf-Ostermann K. Measuring the effectiveness of digital nursing technologies: development of a comprehensive digital nursing technology outcome framework based on a scoping review. BMC Health Serv Res. 2020;20(1):243. https://doi.org/10.11 86/s12913-020-05106-8.

5. Seibert K, Domhoff D, Huter K, Krick T, Rothgang H, Wolf-Ostermann K. Application of digital technologies in nursing practice: results of a mixed methods study on nurses’ experiences, needs and perspectives. Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen; 2020.

6. Alexander GL, Madsen RW, Miller EL, Schaumberg MK, Holm AE, Alexander RL, et al. A national report of nursing home information technology: year 1 results. J Am Med Inform Assoc. 2017;24(1):67–73. https://doi.org/10.1093/ja mia/ocw051.

7. Merda M, Schmidt K, Kähler B. Pflege 4.0 – Einsatz moderner Technologien aus der Sicht professionell Pflegender. Forschungsbericht. Hamburg: Berufsgenossenschaft für Gesundheitsdienst und Wohlfahrtspflege (BGW); 2017.

8. Craig P, Petticrew M. Developing and evaluating complex interventions: reflections on the 2008 MRC guidance. Int J Nurs Stud. 2013;50(5):585–7. https://doi.org/10.1016/j.ijnurstu.2012.09.009.

9. Alaiad A, Zhou L. Patients' adoption of WSN-based smart home healthcare systems: an integrated model of facilitators and barriers. IEEE Trans Prof Commun. 2017;60(1):4–23. https://doi.org/10.1109/TPC.2016.2 632822.

10. Currie LM. Evaluation frameworks for nursing informatics. Int J Med Inform. 2005;74(11–12):908–16. https://doi.org/10.1016/j.ijmedinf.2005.07.007.

11. Fynn JF, Hardeman W, Milton K, Jones AP. A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation. BMC Public Health. 2020;20(1):1000. https:// doi.org/10.1186/s12889-020-09062-0.

12. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108. https:// doi.org/10.1111/j.1471-1842.2009.00848.x.

13. Boll S, Hein A, Heuten W, Wolf-Ostermann K: Grußwort der Organisatoren. In: Zukunft der Pflege Tagungsband der 1 Clusterkonferenz 2018 "innovative Technologien für die Pflege". Edn. Edited by Boll S, Hein a, Heuten W, Wolf- Ostermann K. Oldenburg: OFFIS - Institut für Informatik; 2018: iii-iv.

14. Kowatsch T, Otto L, Harperink S, Cotti A, Schlieter H: A design and evaluation framework for digital health interventions. it - Information Technology 2019;61(5-6):253–63.

15. Vonville H: Screening titles/abstracts, reviewing full text, and reporting results. In: 142nd APHA Annual Meeting and Exposition 2014. New Orleans; 2014.

16. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematicreviews of complex evidence: audit of primary sources. BMJ. 2005; 331(7524):1064–5. https://doi.org/10.1136/bmj.38636.593461.68.

17. Mandelblatt JS, Ramsey SD, Lieu TA, Phelps CE. Evaluating frameworks that provide value measures for health care interventions. Value Health. 2017; 20(2):185–92. https://doi.org/10.1016/j.jval.2016.11.013.

18. Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59. https://doi.org/10.11 86/1471-2288-9-59.

19. Coravos A, Doerr M, Goldsack J, Manta C, Shervey M, Woods B, et al. Modernizing and designing evaluation frameworks for connected sensor technologies in medicine. Npj Digital Medicine. 2020;3(1):37.

20. Haverinen J, Keränen N, Falkenbach P, Maijala A, Kolehmainen T, Reponen J: Digi-HTA: health technology assessment framework for digital healthcare services. Finnish Journal of eHealth and eWelfare. 2019;11(4):326–41.

21. Garcia-Smith D, Effken JA. Development and initial evaluation of the clinical information systems success model (CISSM). Int J Med Inform. 2013;82(6): 539–52. https://doi.org/10.1016/j.ijmedinf.2013.01.011.

22. Khoja S, Durrani H, Scott RE, Sajwani A, Piryani U. Conceptual framework for development of comprehensive e-health evaluation tool. Telemed J E Health. 2013;19(1):48–53. https://doi.org/10.1089/tmj.2012.0073.

23. Broens T, Veld R, Vollenbroek - Hutten M, Hermens H, Halteren A, Nieuwenhuis B. Determinants of successful telemedicine implementations: a literature study. J Telemed Telecare. 2007;13(6):303–9. https://doi.org/10.12 58/135763307781644951.

Krick BMC Nursing (2021) 20:146 Page 18 of 19

24. Chang H. Evaluation framework for telemedicine using the logical framework approach and a fishbone diagram. Healthcare informatics research. 2015;21(4):230–8. https://doi.org/10.4258/hir.2015.21.4.230.

25. Lau F, Hagens S, Muttitt S. A proposed benefits evaluation framework for health information systems in Canada. Healthcare Quarterly. 2007; 10(1):112–6.

26. Sockolow PS, Crawford PR, Lehmann HP. Health services research evaluation principles. Broadening a general framework for evaluating health information technology. Methods Inf Med. 2012;51(2):122–30. https://doi. org/10.3414/ME10-01-0066.

27. Sadoughi F, Kimiafar K, Ahmadi M, Shakeri MT. Determining of factors influencing the success and failure of hospital information system and their evaluation methods: a systematic review. Iran Red Crescent Med J. 2013; 15(12):e11716.

28. Stylianides A, Mantas J, Roupa Z, Yamasaki EN. Development of an evaluation framework for health information systems (DIPSA). Acta Inform Med. 2018;26(4):230–4. https://doi.org/10.5455/aim.2018.26.230-234.

29. Yusof MM. A socio-technical and lean approach towards a framework for health information systems-induced error. Stud Health Technol Inform. 2019;257:508–12.

30. Rouleau G, Gagnon MP, Côté J, Payne-Gagnon J, Hudson E, Dubois CA. Impact of information and communication technologies on nursing care: results of an overview of systematic reviews. J Med Internet Res. 2017;19(4):e122.

31. Kidholm K, Bowes A, Dyrehauge S, Ekeland AG, Flottorp SA, Jensen LK, Pedersen CD, Rasmussen J: The MAST Manual. MAST - Model for ASsessment of Telemedicine. In.: MethoTelemed team; 2010.

32. Mathews SC, McShea MJ, Hanley CL, Ravitz A, Labrique AB, Cohen AB. Digital health: a path to validation. NPJ Digit Med. 2019;2(1):38. https://doi. org/10.1038/s41746-019-0111-3.

33. Poulin P, Austen L, Scott CM, Waddell CD, Dixon E, Poulin M, et al. Multi- criteria development and incorporation into decision tools for health technology adoption. J Health Organ Manag. 2013;27(2):246–65. https://doi. org/10.1108/14777261311321806.

34. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res. 2017; 19(11):e367. https://doi.org/10.2196/jmir.8775.

35. Bakken S, Ruland CM. Translating clinical informatics interventions into routine clinical care: how can the RE-AIM framework help? J Am Med Inform Assoc. 2009;16(6):889–97. https://doi.org/10.1197/jamia.M3085.

36. Van Der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc. 2003;10(3):235–43. https://doi.org/10.1197/jamia.M1094.

37. Yusof MM, Papazafeiropoulou A, Paul RJ, Stergioulas LK. Investigating evaluation frameworks for health information systems. Int J Med Inform. 2008;77(6):377–85. https://doi.org/10.1016/j.ijmedinf.2007.08.004.

38. Vis C, Bührmann L, Riper H, Ossebaard HC. Health technology assessment frameworks for eHealth: a systematic review. Int J Technol Assess Health Care. 2020;36(3):204–16. https://doi.org/10.1017/S026646232000015X.

39. Bradford N, Chambers S, Hudson A, Jauncey-Cooke J, Penny R, Windsor C, et al. Evaluation frameworks in health services: an integrative review of use, attributes and elements. J Clin Nurs. 2019;28(13–14):2486–98. https://doi. org/10.1111/jocn.14842.

40. Nguyen H, Saranto K, Tapanainen T, Ishmatova D. A review of health information technology implementation success factors: importance of regulation and finance; 2014.

41. Lau F. Extending the infoway benefits evaluation framework for health information systems. Stud Health Technol Inform. 2009;143:406–13.

42. McNair J, Ammenwerth E, Nykänen P, Talmon J. Factors influencing success and failure of health informatics systems: a pilot Delphi study. Methods Inf Med. 2006;45:125–36.

43. Greenhalgh T, Maylor H, Shaw S, Wherton J, Papoutsi C, Betton V, Nelissen N, Gremyr A, Rushforth A, Koshkouei M, Taylor J. The NASSS-CAT Tools for Understanding, Guiding, Monitoring, and Researching Technology Implementation Projects in Health and Social Care: Protocol for an Evaluation Study in Real-World Settings. JMIR Res Protoc. 2020;9(5):e16861. https://doi.org/10.2196/16861.

44. Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in

systematic reviews: a methodological systematic review. BMC Med Res Methodol. 2019;19(1):132. https://doi.org/10.1186/s12874-019-0782-0.

45. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50. https://doi.org/10.1186/1748-5908-4-50.

46. Wahlster P, Brereton L, Burns J, Hofmann B, Mozygemba K, Oortwijn W, Pfadenhauer L, Polus S, Rehfuess E, Schilling I et al: Guidance on the integrated assessment of complex health technologies – the INTEGRATE- HTA model. 2016.

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Krick BMC Nursing (2021) 20:146 Page 19 of 19

BioMed Central publishes under the Creative Commons Attribution License (CCAL). Under the CCAL, authors retain copyright to the article but users are allowed to download, reprint, distribute and /or copy articles in BioMed Central journals, as long as the original work is properly cited.