SYSTEMATIC REVIEWS AND META-ANALYSES

Contents lists available at ScienceDirect

Journal of Professional Nursing

journal homepage: www.elsevier.com/locate/jpnu

Development of a DNP measurement grid to increase the rigor of doctor of nursing practice students' data collection and analysis methods Linda L. Costaa,⁎, Debra Binghama,b, Carla L. Storra, Margaret Hammerslaa, Jeffrey Martina, Charlotte Seckmana aUniversity of Maryland School of Nursing, 655 West Lombard St., Baltimore, MD 21201, United States of America b Institute for Perinatal Quality Improvement, 377 Willard Street, #284, Quincy, MA 02169, United States of America

A R T I C L E I N F O

Keywords: Doctor of nursing practice Doctoral education Measurement Data collection Data analysis

A B S T R A C T

A key component of the DNP project rigor is the collection and analysis of data or measurement. A Steering Committee at the University of Maryland formed to improve the quality of DNP projects established a workgroup to evaluate the current measurement content in four DNP core courses with the goal of establishing DNP project measurement criteria across the curriculum. The steps included: Step 1: Identify QI Measurement Methods and Tools. Identify the essential QI measurement methods and tools recommended by national organizations. Step 2: Create a DNP Measurement Grid. Define main data methods topics with subtopics. Step 3: Map the DNP core courses. Using the DNP Measurement Grid criteria determine the measurement content included in each course and student mastery level. The level of mastery was ranked from introduced (awareness), to reinforced (knowledge), to demonstrated (application). Step 4: Evaluate and Refine the DNP Measurement Grid Criteria. Adjustments were made in the DNP curriculum to include topics and subtopics at the desired mastery level. The rigor of data measurement and analysis will be evaluated in future DNP projects. The workgroup's four-step process provides a path that facilitated improving curriculum measurement content. This process may provide guidance for others undertaking similar work.

Introduction

Nursing doctoral programs prepare clinical and research leaders to advance the practice of nursing and improve health locally and glob- ally. Whereas a Doctor of Philosophy (PhD) is a degree designed to prepare individuals to conduct research studies or in other words generate empirical evidence that advances health care, a Doctor of Nursing Practice (DNP) degree is generally understood to be a degree that prepares nurses to obtain advanced clinical practice, expertise in leadership, implementation science, and translation of research to practice (AACN, 2006; Embree et al., 2018; Root et al., 2018). Practice scholars are expected to be knowledgeable of methods and tools that can effectively reduce the time to translate knowledge to practice (Institute of Medicine, 2001) and improve care processes (AACN, 2018). National organizations such as the American Association of Colleges

of Nursing's: Essentials of Doctoral Education for Advanced Nursing Practice, Essential III - “Clinical Scholarship and Analytical Methods for Evidence-Based Practice” outlines the knowledge and skills a DNP

program is supposed to cover in their curriculum in order to prepare DNP graduates to effectively translate research into practice (AACN, 2006). Specifically AACN states that the knowledge and skills that DNP graduates need include the ability to “design and implement processes to evaluate outcomes of practice, practice patterns, and systems of care within a practice setting, health care organization, or community against national benchmarks to determine variances in practice out- comes and population trends”, “design, direct, and evaluate quality improvement methodologies to promote safe, timely, effective, effi- cient, equitable, and patient-centered care”, “analyze data from prac- tice”, “predict and analyze outcomes, and examine patterns of behavior and outcomes” (AACN, 2006, pg. 12). Other national organizations have published competency statements that outline the need for ad- vanced practices nurses to be proficient in the translation of research into practice (Council on Accreditation of Nurse Anesthesia Educational Programs, 2009; National Organization of Clinical Nurse Specialists, 2009; NONPF, 2002a, 2002b; QSEN Institute, 2020). However, none of these organizations include a list of the specific data and measurement knowledge and skills DNP students need to master prior to graduation.

https://doi.org/10.1016/j.profnurs.2020.09.006 Received 1 May 2020; Received in revised form 4 September 2020; Accepted 11 September 2020

⁎ Corresponding author. E-mail addresses: [email protected] (L.L. Costa), [email protected] (D. Bingham), [email protected] (C.L. Storr),

[email protected] (M. Hammersla), [email protected] (J. Martin), [email protected] (C. Seckman).

Journal of Professional Nursing 36 (2020) 666–672

Available online 14 September 2020 8755-7223/ © 2020 Elsevier Inc. All rights reserved.

T

Measurement is a powerful tool for assessing whether changes being made are actually leading to improvement and it is a key component of quality improvement (QI) projects. In QI initiatives, measurement en- compasses a plan to develop and use clinical quality measures. A DNP graduate should be expected to understand the quality of a measure by not only evaluating the data source and how the data was collected, but also the suitability and properties (e.g., benchmarking-endorsed by the National Quality Forum), and the relevancy to their targeted consumer. In addition, DNP graduates should be especially strong in the health- care related measurement skills of accountability and improvement (Ogrinc et al., 2018). Measurement is important to accountability so one can quantitatively assess implementation and outcomes particu- larly if national measures and benchmarks have been established. Comparing outcomes of aggregate data between, hospitals, and states enables one to evaluate whether performance is better or similar to other units. For example, the State of Maryland's Health Services Cost Review Commission (HSCRC) (2020) works with hospitals in the state to set benchmarks for performance that are linked with achieving fi- nancial rewards or penalties. Similar to CMS, components of evaluation include hospital-acquired conditions, potentially avoidable read- missions, and quality-based reimbursement. Understanding strategic QI priorities of organizations impacting revenues is critical for DNP graduates. Practice sponsors allocating scarce resources at clinical sites should set a higher priority to implement data-driven projects. The rapid proliferation of DNP programs in the past 10–15 years has

led to a wide variation in curriculum and program objectives and outcomes (Grey, 2013), and no consensus on how to define rigor of a DNP program (Root et al., 2018). In an evaluation of the rigor of 35 of 40 DNP projects posted by “recent graduates” via an online portal (five projects were not reviewed due to planned publication by students), only 18% of the projects were felt to have met the criteria for doctoral level scholarship (Root et al., 2018). An examination of rigor and value of 65 final DNP scholarly projects selected from online repositories from 42 different schools found only 13 projects earned a score equal to or above 75% on the DNP Project Critical Appraisal Tool (DNP-PCAT) (Roush & Tesoro, 2018). Evaluations of the implementation process and health system outcomes were found to be lacking. These findings are consistent with concerns published by others that DNP projects are not scholarly or rigorous enough for doctoral level education (VanderKooi et al., 2018; Waldrop et al., 2014). Quality improvement methods have been proposed as a way to help increase the scholarship and rigor of DNP students' projects (Sun & Cherry, 2018; Terhaar & Sylvia, 2015). Quality improvement has been defined as “systematic, data guided activities designed to bring about immediate, positive changes in the delivery of health care in particular settings” (Baily et al., 2006, p.S5). Thus, it is our belief that data-guided DNP QI projects are likely to be more rigorous than projects that have insufficient measurement plans to track changes over time in structures, processes, and outcomes. This paper describes the process of an internal evaluation of the

DNP program curriculum undertaken by University of Maryland's School of Nursing's (UMSON) Measurement Workgroup to develop measurement topics that would lead to competency in data-driven QI work expected of the DNP graduate. We share a DNP Measurement Grid with applied definitions developed by the workgroup and the re- commendations for its use.

Methods

Under the direction of the UMSON's Advancing Implementation Science Education (AdvISE) Steering Committee, which started working in 2017, and received grant support from the Maryland Higher Education Commission in July 2018. The AdvISE Steering Committee agreed upon a vision statement and completed a detailed review of the drivers of the DNP students' project success was undertaken. Primary drivers include faculty and student expertise, curriculum, and health- care partnerships. Primary drivers were further defined by secondary

drivers. Secondary drivers for the curriculum included, the students' DNP projects, core supports e.g. courses, clinicals/practicums, and specialty. The resulting thorough review of the challenges allowed the Steering Committee to come to consensus on the priorities for review and action and the formation of four workgroups: Theory, Leadership, Measurement, and DNP Project Coursework. Each workgroup worked independently and collaboratively to improve the DNP curriculum, partnerships, and the rigor of DNP projects by addressing each of the primary and secondary drivers. More information about the AdvISE effort is available online (https://www.nursing.umaryland.edu/news- events/events/advise/). The goal of the Measurement Workgroup was to close curriculum QI

gaps by adding QI measurement concepts, methods, and support for students that would guide them in defining measures, collecting, ana- lyzing and presenting data results. The stated outcome was that DNP graduates would be competent to examine and translate data to address clinical problems and show a high level of scholarship when dis- seminating their findings. One major priority was an initial focus on further assessing and improving QI methods and measurement content in the DNP coursework. The measurement workgroup was led by one of the steering com-

mittee members. Other steering committee members with expertise in measurement were joined by the course directors from our school's four core DNP courses. These core courses provide an intersection where two role categories for DNP students merge: 1) Specialization as an APN with focus on care of individuals and 2) Practice at a systems or orga- nizational level (AACN, 2006). Therefore, the core courses were se- lected as ideal for curriculum analysis of content on QI methods and measurement. A description of the core courses is found in Table 1. The procedure to document measurements for the DNP curriculum

consisted of four sequential steps. Step 1: Identify QI Measurement Methods and Tools. A list of QI

measurement topics and subtopics were identified based on re- commendations from the Institute for Healthcare Improvement (IHI) (Ogrinc et al., 2018), the Quality and Safety Education in Nursing (QSEN) (QSEN Institute, 2020), and QI experiences of the workgroup members. Step 2: Create a DNP Measurement Grid. The initial list of mea-

surement topics was narrowed down based on consensus. The main measurement topics were: quality measure types, QI measurement considerations, data visualization, data collection considerations, and data sources. Measurement skills and concepts under every main topic began to emerge as subtopics (See Table 2 for topics/subtopics and applied definitions). Each subtopic was ranked by the required mastery level that the workgroup proposed appropriate for our school's DNP students. Three levels of mastery were identified: Level 1- students needed to demonstrate application of topic in their DNP project, Level 2 - students needed to demonstrate knowledge of the topic in their course work, but was not required in their DNP project, and Level 3- students express awareness and recognition. Step 3: Map the DNP Courses. The DNP Measurement Grid was

further refined as we identified what QI content existed in each of the DNP courses. The course mapping took into account the current level of instruction by course: (I = introduction (overview), R = reinforced (applied, integrated examples) and where comprehension is demon- strated (D) via exam or exercise). The resulting DNP Measurement Grid enabled group members to see where deficit areas (e.g. lack of pro- gression from introduction to reinforced) were located and identify opportunities to include measurement content in course curriculum revisions. At this point, the baseline state was identified demarcating the content of the course before the measurement workgroup com- menced (baseline, B). Then, changes made up to Fall 2019 as a result of workgroup discussion are indicated (C). This designation allowed workgroup members to track progress of the addition of measurement concepts into the core courses. The template for course mapping of the identified key quality improvement concepts is shown in Table 3.

L.L. Costa, et al. Journal of Professional Nursing 36 (2020) 666–672

667

The workgroup took into consideration that course changes would only effect future cohorts of students, so steps were also taken to ad- dress the needs of current students who were already further along in their course sequence. Additional adjunct resources were developed to enhance the skills and knowledge of approximately 75 faculty members who teach the approximately 500 students who are currently enrolled in UMSON's DNP program. One strategy we found helpful for sup- porting faculty and students was to develop on-demand educational videos (“QI Briefs”) which not only filled an immediate gap, but in the future would also serve as a simple refresher for faculty supporting students and for students planning their DNP projects. These educa- tional videos include topics such as measurement in QI projects, run charts, developing logic models, and dissemination of DNP QI projects. The videos are available to students and faculty on the school's edu- cational technology website. These videos have been popular and helpful for their specific focus and brevity. They also are associated with links to tool templates and additional industry references for fur- ther study as desired. Step 4: Evaluation of and Refine the DNP Measurement Grid

Criteria. The DNP Measurement Grid is a living document, meaning we can adjust and modify the grid as needed as we expand the conversation about our work through publication and presentations. For example, in September 2019 the UMSON hosted the one-day AdvISE Summit that brought together national experts to discuss how to expand im- plementation science and quality improvement capacity among doc- toral faculty and students. At the AdvISE Summit (2019), a panel of QI experts and implementation scientists were asked to comment on the workgroup's DNP Measurement Grid. This Expert Panel provided sug- gestions for future steps and refinement of the grid.

1. Define your guiding quality question. Ensure the question is im- portant to key stakeholders.

2. Design a measurement plan that collects enough data to evaluate improvement and minimize the measurement burden.

3. Items ranked with mastery level 1 need to be re-evaluated.

The discussion at the Summit provided valuable feedback from the expert panel and attendees from across the country validating the need for ongoing refinement of the expectation for measurement in the DNP program. The AdvISE Summit video can be accessed at: AdvISE Summit Video Summary (2020). https://youtu.be/yUc4UiNq9A8.

Discussion

We believe that one important way to close the evidence to practice gap is to enhance the measurement skills DNP graduates possess when planning and tracking the implementation of evidence-based practices. We found that there was limited educational content in our curriculum about data and measurements beyond the biostatistics course. This identified need led us to make additions and improvements to the DNP curriculum. Refinement of the DNP Measurement Grid and course curricula is an ongoing process as the expectation of the DNP graduate has changed overtime, and also varies by employer and role. Interviews with twenty-three employers found that DNP prepared nurses had an impact on organization outcomes through review of clinical data and translating research into practice solutions (Beeber et al., 2019). Measurement is a cornerstone of improvement. Analyzing system

processes over time evaluate efficiency and effectiveness of achieving quality improvement aims. A balanced set of measures are needed for all measurement work, including structure, process, outcome and bal- ancing measures (when appropriate). Students are required to include structure, process, and outcome measure in their master measurement plan. The data plan is created over two pre-implementation semesters, and students are encouraged to use nationally recognized measures. QI measurement considerations include the use of national bench-

marks. The National Quality Forum (NQF) is the organization that is responsible for endorsing QI metrics in the United States (http://www. qualityforum.org/Field_Guide/List_of_Measures.aspx). One of the chal- lenges to expanding the use of QI data metrics among DNP students is that there are not validated and NQF endorsed structure, process, and outcome measures for all of the evidence-based interventions that the DNP students are implementing. This means that the faculty and

Table 1 Core courses and description.

Course Description

Methods for research and EBP In this core course, the student develops competencies to identify practice issues, retrieve and critically appraise research, and synthesize sources of evidence to engage in decision making for evidence-based practice change. Research and analytical methods are applied to evaluate the development of best practices to improve patient outcomes. Students demonstrate competencies by developing a PICOT(T), constructing an Evidence Review Table, and synthesizing research and other sources of evidence to make a recommendation for practice. Comparative analysis of Quality Improvement and Research is addressed with regards to the purpose, methods, outcomes, and use of each.

Biostatistics for EBP This core course extends basic statistics competencies by providing a working knowledge of common descriptive and inferential statistics used in evidence-based practice, including chi-square, t-test, ANOVA, correlation, and regression. One module is devoted to introducing Quality Improvement measures and charts. An applied approach is taken where students conduct analyses using illustrative datasets and common analytical software to practice generating, evaluating, and using evidence. Accurate and concise reporting of results in text, tabular, and graphical form is emphasized.

Information systems for improvement and technology for the improvement and transformation of health

This core course designed to provide the DNP student with the knowledge and skills necessary to utilize healthcare technologies and resources to support practice, quality improvement initiatives, and administrative decision making in order to improve and transform healthcare. The didactic course focuses on an advanced understanding and application of informatics theories and performance improvement models. In a 1- credit practicum the student applies concepts and principles from the didactic course in a work setting. Students create a learning contract and are provided guidance on the practice of deriving data, information, and knowledge from existing healthcare information systems that will be used to support and improve patient care

Translating evidence to practice This core course focuses on how to translate, evaluate and disseminate evidence in a contemporary health care environment. Individual, organizational and global barriers to translating evidence into practice are explored. The Knowledge to Action Framework is used as the course model. The DNP student identifies a practice problem, explores the gap between current practice and best practice, and completes an evidence synthesis. Knowledge translation (KT) tools are identified e.g. clinical practice guidelines. Knowledge is tailored as the action cycle begins with a cultural assessment of the practice area. Clinical and economic outcomes measures of the knowledge translation project are defined. A knowledge translation framework guides the student's translation plan.

L.L. Costa, et al. Journal of Professional Nursing 36 (2020) 666–672

668

Table 2 DNP measurement grid applied definitions.

Topics/subtopics Mastery levela Applied definition

I. Quality measure types A balanced set of measures for all improvement efforts Structure 2 Organization's ability to provide quality care. Examples of structures are education offerings,

electronic health records, the physical layout of a health facility, the location of the health facility, staffing models, length of orientation.

Process 1 Behaviors of the clinical team when providing care and interactions between practitioners and patients. Examples of processes are when, where, and how a clinical intervention, e.g., a flu vaccine, blood or medications, are administrated to a patient

Outcome 1 Change in health status. Balancing 2 Do changes designed to improve care cause new problems

II. QI measurement considerations Identifying aspects of measures to capture data that will provide value. Goal setting/benchmarking 1 Nationally Endorsed Measures. The National Quality Forum (NQF) is the organization that is

responsible for endorsing measures in the United States. NQF's process ensures that measures are reliable, validated, feasible, and important.

What to do if no nationally endorsed measures 1 Identify the best evidence available regarding which measures should be utilized for the QI project. Refer to research published that supports the implementation of the evidence-based structures, processes, and outcomes. Utilize this research evidence to justify how the numerator and denominators were determined, e.g., the justification for the inclusion and exclusion criteria.

Variation in QI and risk adjustment considerations 1 Variation is unintended and not desirable when the population of interest does not receive the desired evidence-based intervention (structures and/or processes) that have been shown to lead to desired outcomes. Variation is often measured using statistical process control charts and run charts. Populations can be risk stratified in order to compare populations that are most alike. For example, a common risk stratification for comparing different types of cesarean births rates is to create a sub- group of women who are considered at low-risk for having a primary cesarean birth. This sub-group includes first birth mothers, whose babies are at term gestation, and whose baby presents in the desired head down position (nulliparous, term, singleton, vertex – NTSV). By creating this sub-group, the QI leader can compare rates across regions, counties, hospitals, etc. to determine where and how much variation exists. This information provides insights into which birthing facilities have the highest rates of NTSV cesarean birth rates and require the most support to eliminate overuse of cesarean births. However, risk adjustments models need to be applied from a racial equity perspective. Specifically, variation should not be eliminated by creating a risk adjustment model that hides disparities in outcomes due to race and ethnicity.

Population selection - include vulnerable populations 1 QI initiatives are designed to improve structures, processes, and outcomes. Vulnerable populations must be included in QI initiatives. Additional efforts may be needed to ensure that evidence-based interventions are implemented in an equitable manner from a racial and social justice perspective.

Identifying the numerator & denominator (inclusion and exclusion criteria)

1 The numerator is the measure focus describing the target process, condition, event, or outcome expected for the targeted population; The denominator defines the population being measured—it could be the whole population or a subset; The denominator exclusion identifies members of this population who should be removed from the measure population, and hence the denominator, before determining if numerator criteria are met.

Comparing pre- and post-data differences 2 Further definition of the effect of QI interventions and outcomes Sample size 2 Enough data to determine change and understand variation.

III. Data visualization The graphical representation of information and data. Variation (e.g., boxplots) 1 Standardized ways to display the distribution of data Creating & interpreting run charts 1 Display observed data over time in line graphs, allows one to detect trends or patterns in the process Other methods to display overtime 2 Visually track processes and abnormalities. Statistical Process Control charts that analyze process

performance looking for special vs common cause. IV. Data collection considerations Collection aspects/methods to heed in order to obtain quality data and to be able to detect changes Sampling/auditing electronic medical records 1 Approach to choosing which records to include (systematic or random) Data timing 1 When to collect data -should be regularly collected (weekly, monthly or quarterly, depending on

needs) as a part of the monitoring process. This enables swift interventions and countermeasures to be implemented as required

Data granularity 1 The level of detail of the data within the data structure. The greater the granularity, the deeper the level of detail (data in pieces) which makes it easier for analysts to mine and analyze.

Type of variables/level of measurement 1 Role of measure (e.g. independent/dependent, confounder). Classification that describes the nature of information within the values assigned to variables (nominal, ordinal, interval/ratio; dichotomous vs continuous)

Feasibility/missing data 1 Awareness of design features that impact whether data collection can be done as one plans (feasibility) and obtain the information needed (avoid missing values)

Focus group 3 A data collection method that brings small groups of people together to provide feedback. A moderator leads a discussion to obtain useful information

Observation 3 Method by which one gathers knowledge of the researched phenomenon through making observations of the phenomena, as and when it occurs.

Self-report (interview/survey) 3 Methods that capture responses provided by the participant to questions posit to them. Respondents indicate/select a response by themselves without interference.

Electronic collection methods 2 Use of electronic devices to generate health data, e.g., Apps, medical devices designed to collect information directly from the patient

Electronic data extraction 2 Downloading of data related to specific variables from large data sources. Data analysts know data elements and can download if approved by organization.

V. Data sources Data that track quality, safety, and utilization of health care services that are available in databases Electronic health record 2 Administrative systems and encounter data: Hospital administrative databases that capture

demographic data, sources/type of admission, clinical data and notes. Financial charges and expenses, payor data.

(continued on next page)

L.L. Costa, et al. Journal of Professional Nursing 36 (2020) 666–672

669

students require support in identifying the structure, process, and out- come metrics that they will be using. In particular, it has been chal- lenging at times to determine what their QI metric's numerator and denominators should be and what should be the exclusions. For

example, if a DNP student is working to increase an evidence-based process such as the use of immediate and continuous skin-to-skin after a woman gives birth, the student needs to identify which women and newborns are in the population of interest (the denominator) and how to track and calculate which women and infants received the inter- vention (numerator). At this point, despite years of research outlining the physiologic benefits of immediate and continuous skin-to-skin and the completion of feasibility and reliability testing for these two me- trics, immediate and continuous skin-to-skin QI metrics have not gone through validity testing and are not yet eligible to be reviewed for endorsement by NQF (Bingham et al., 2019). Students and faculty should not wait to implement skin-to-skin, since the evidence-base supporting this practice is well known. However, the student and fa- culty will need support in determining the specific inclusion and ex- clusion criteria for these and other QI structure, process, and outcome metrics. At UMSON we require students to find published information about the QI metrics they include in their data plan and to use na- tionally endorsed and validated measures when possible. Data visualization provides a graphic representation of data. QI

data-are often displayed over time using run charts or statistical process control charts because QI leaders need more real-time information to determine whether their implementation strategies and tactics are ef- fective or not (Ogrinc et al., 2018). Using pre- and post-implementation data collection methods limited students' ability to fully appreciate the normal operating rhythm, or track (closer to real-time) whether their implementation strategies and tactics resulted in desired improvement and maintained a process within a desired range. Beginning Spring 2020, all graduating students are required to present their process data in run charts. This is a major change for the faculty as well, as most of them had no quality improvement training or proficiency in visual data displays. Requiring all students to demonstrate proficiency in run charts is also helping us identify other ways in which the rigor of the students' DNP project could be strengthened. For example, in order to develop a useful measurement plan, students need to be very clear about the evidence-based improvement that they are implementing, i.e. clearly documenting and validating the problem statement, scope, and goals. Providing clarity and allowing the DNP student to map how to oper- ationalize their project and measurement plan enabled faculty to pro- vide timely feedback to strengthen the project plan. For the DNP stu- dent use of run charts required additional education regarding the understanding of variation, to strengthen students understanding of the need to plan their measurement path to have adequate data points for analysis (Ogrinc et al., 2018). Data collection considerations include identifying measures to

capture data that provide value including data timing and data gran- ularity. The most effective data used in QI is real-time data. The older the data, the harder it is to know whether there have been changes in structures and processes. For example, if the long-term goal is to eliminate preventable falls and injuries from falls and the data is more than 6 months old, then structures and processes may have changed since these data were collected. In addition, having data that is granular or having data about the specific behaviors or processes of individual clinicians also supports the implementer's efforts to identify which

Table 2 (continued)

Topics/subtopics Mastery levela Applied definition

Claims and ancillary coding 3 ICD 10 diagnoses codes reflect specificity by body system, Current Procedural codes, (CPT) procedure and medical services, ER/outpatient; APR-DRG with severity, pediatric and OB. Pharmacy drug codes; procedure/billing codes Medication usage and formulary compliance

Population based data sources 2 National/State/City databases-population representative samples. Date (Timing) must be carefully noted. Healthy People 2020 goals.

Other data sources 2 Patient generated health data (e.g. social media, internet); Personal Health Records (PHR); Medical Device registries;

a Mastery level ratings: 1 = students need to demonstrate in their project (application), 2 = students need to demonstrate in their course work, but not their DNP project (knowledge), and 3 = students need to comprehend the concept or skill (awareness).

Table 3 DNP measurement grid template for course mapping of quality improvement concepts.

Courses

Masterya 1 2 3 4

Bb C B C B C B C

Quality measure types Structure Process Outcome Balancing

QI measure considerations Goal setting/benchmarking What to do if no nationally endorsed

measures Variation in QI and means, risk

adjustment considerations Population selection - include vulnerable

populations Identifying the numerator &

denominator (inclusion and exclusion criteria)

Comparing pre- and post-data differences

Sample size

Data visualization Variation (e.g., boxplots) Creating & Interpreting run charts Other methods to display overtime

Data collection considerations Sampling/auditing electronic medical

records Data timing Data granularity Type of variables/level of measurement Feasibility/missing data Observation Self-report (interview/survey) Electronic collection methods Electronic data extraction

Data sources Electronic health record (EHR) Claims and ancillary coding Population based data sources Other data sources

a Mastery level ratings: 1 = students need to demonstrate in their project (application), 2 = students need to demonstrate in their course work but not their DNP project(knowledge), and 3 = students need to comprehend the concept or skill (awareness). b B = content in course prior to<date> , C = changes made in course

content by< date> .

L.L. Costa, et al. Journal of Professional Nursing 36 (2020) 666–672

670

implementation strategies will be the most effective. For example, an implementer will make different choices if only three clinicians in the group are not following the new falls protocols and all of the falls that occur happen when they are on duty than if the falls are spread evenly across all clinicians and shifts. Waiting until the project ended to de- termine implementation success was a poor alternative given the se- mester-based time constraint of the project. It has been common for students to learn of critical short comings in their improvement plans and data collection methods too late to make project adjustments. Instead, students need to plan for collection of timely data to help them adjust and re-adjust their approaches as needed (e.g. running multiple Plan-Do-Study-Act cycles). Our focus for the DNP students is not only on teaching them to study whether an intervention was effective or not (projects are evidence-based), we focus on whether an evidence-based intervention was adopted within a particular context (a hospital unit, a clinic setting). Providing students’ information about data sources encourages

them to explore possible measures pertinent to their project. Data sources are defined as including specific measures of health care quality that use available databases. The use of data from EHR to evaluate healthcare outcomes is important to the scholarship of practice (AACN, 2018). Data sources need to be evaluated for completeness, correctness, concordance, plausibility, and currency (Weiskopf & Weng, 2013). These data elements were defined and collected by others. Therefore, the use of the data needs to be evaluated with this in mind. When working with a data analyst, the DNP student needs to provide the clinical practice question, the sample population to select (International Classification of Diseases (ICD-10-CM), Current Procedural Termi- nology (CPT) codes, All Patients Refined Diagnosis Related Groups (ARP-DRG)); the time period, summary data or individual level and specific variables for measurement. Data used for the DNP project will vary depending on the project. The DNP graduate should be aware of secondary data sources found in national databases such as the National Center for Health Statistics, the Centers for Disease Control (CDC), and Centers for Medicare and Medicaid (CMS) which may be valuable either now or as they enter the workforce as graduates. One of most exciting improvements made to our Translating

Evidence to Practice course was to reinforce the measure: Electronic data extraction. New didactic content, “Leveraging Electronic Health Record (EHR) Data for EPB and Quality Improvement” was added to inform students not only of the specificity needed for data requests and sources of electronic data but also to recognize priority process for data requests in organizations. Some previous DNP students had expected new data elements for downloading their project results and ended up having to complete a manual audit of the EHR to collect project data. This synchronous session allowed students to discuss their experiences in clinical practice initiating data requests and the results of lack of specificity resulting in unusable data from the EHR download. The results of the workgroup analysis and input from experts have

been enlightening. Our next step is to evaluate the DNP students' pro- jects for measurement and data quality and rigor by reviewing the final papers and posters. This review will take two to three years for student cohorts exposed to new core components to complete their DNP their program as the project is completed at the end of the program.

Limitations

The information presented is limited to the curriculum of one DNP program. Identified measurement categories are from an internal eva- luation and consensus of the. Institution's DNP program AdvISE measurement workgroup. We are

not aware of any way to systematically compare our experiences with those of other Universities to determine the generalizability of our work.

Conclusions

We have made strides toward expanding measurement content throughout our DNP program curriculum; however, more work is needed. We share our progress at this stage with the idea that our journey may be useful to others. The DNP Measurement Grid and de- velopment process could be useful for other schools who wish to evaluate their DNP courses. As others work on the processes of building measurement knowledge, we welcome discussion and input on the list of measurement topics we created. The aim of QI is to improve health care and we continue to explore what is the appropriate level of rigor for DNP project work.

Acknowledgment

The authors wish to acknowledge the contributions and support of other members of the Advancing Implementation Science Education (AdvISE) Steering Committee members Kathleen Buckley, PhD, RN, Shannon Idzik, ANP-BC, FAANP, FAAN, Susan Bindon, DNP, RN, NPD- BC, CNE, CNE-cl, Renee Franquiz, DNP, RN, Kathryn Montgomery, PhD, RN, NEA-BC, Patricia Franklin, PhD, RN, and Lucy-Rose Davidoff, RN. Part of this work was funded by the Nurse Support Program II,

which in turn is funded by the Maryland Health Services Cost Review Commission and administered by the Maryland Higher Education Commission, grant number NSP II 19-1.

References

AdvISE Summit (2019). https://www.nursing.umaryland.edu/news-events/events/ advise.

AdvISE Summit Video Summary (2020). https://youtu.be/yUc4UiNq9A8. American Association of Colleges of Nursing (2018). Defining scholarship for academic

nursing: Task force consensus position statement. https://www.aacnnursing.org/ Portals/42/News/Position-Statements/Defining-Scholarship.pdf.

American Association of Colleges of Nursing ACN (2006). The essentials of doctoral education for advanced nursing practice. https://www.aacnnursing.org/Portals/42/ Publications/DNPEssentials.pdf.

Baily, M. A., Bottrell, M., Lynn, J., & Jennings, B. (2006). Special report: The ethics of using QI methods to improve health care quality and safety. The Hastings Center Report, 36(4), S1–40. https://doi.org/10.1353/hcr.2006.0054.

Beeber, A. S., Palmer, C., Waldrop, J., Lynn, M. R., & Jones, C. B. (2019). The role of doctor of nursing practice-prepared nurses in practice settings. Nursing Outlook, 67(4), 354–364. https://doi.org/10.1016/j.outlook.2019.02.006.

Bingham, D., Boisvert, M. E., Webb, A., & Muri, J. (2019). Feasibility of AWHONN’s immediate and continuous skin-to-skin nursing care quality measures. Journal of Obstetric, Gynecologic, and Neonatal Nursing, 48(5), 516–525.

Council on Accreditation of Nurse Anesthesia Educational Programs (2009). Position statement on doctoral education for nurse anesthetists. Retrieved from https://www. coacrna.org/about-coa/position-statements/.

Embree, J. L., Meek, J., & Ebright, P. (2018). Voices of chief nursing executives informing a doctor of nursing practice program. Journal of Professional Nursing, 34(1), https:// doi.org/10.1016/j.profnurs.2017.07.008.

Grey, M. (2013). The doctor of nursing practice: Defining the next steps. Journal of Nursing Education, 52(8), 462–465. https://doi.org/10.3928/01484834- 20130719-02.

Institute of Medicine (2001). Crossing the quality chasm: A new health system for the 21st

century. National Academies PressMaryland HSCRC. 2020 https://hscrc.maryland. gov/Pages/hscrc-workgroup-performance-measurement.aspx.

National Association of Clinical Nurse Specialists (2009). Position statement on the nursing practice doctorate. Retrieved from http://www.nacns.org/docs/ PositionOnNursingPracticeDoctorate.pdf.

National Organization of Nurse Practitioner Faculties (2002a). Statement from the NONPF Board of Directors. Retrieved from http://www.nonpf.org/?page=83.

National Organization of Nurse Practitioner Faculties (2002b). Clinical doctorate in- itiative. Retrieved from http://www.nonpf.org/?page=83.

Ogrinc, G., Headrick, L., Barton, A., Dolansky, M., Madigosky, W., & Miltner, R. (2018). Fundamentals of health care improvement (3rd ed.). Oak Brook Illinois: Joint Commission Resources.

QSEN Institute (2020). https://qsen.org. Root, L., Nuñz, D. E., Velasquez, D., Malloch, K., & Porter-O’Grady, T. (2018). Advancing

the rigor of DNP projects for practice excellence. Nurse Leader, 16(4), 261–265. https://doi.org/10.1016/j.mnl.2018.05.013.

Roush, K., & Tesoro, M. (2018). An examination of the rigor and value of final scholarly projects completed by DNP nursing students. Journal of Professional Nursing, 34(6), 437–443. https://doi.org/10.1016/j.profnurs.2018.03.003.

Sun, G. H., & Cherry, B. (2018). Using the logic model framework to standardize quality

L.L. Costa, et al. Journal of Professional Nursing 36 (2020) 666–672

671

and rigor in the DNP project. Nurse Educator, 44(4), 183–186. https://doi.org/10. 1097/NNE.0000000000000599.

Terhaar, M. F., & Sylvia, M. (2015). Scholarly work products of the doctor of nursing practice: One approach to evaluating scholarship, rigour, impact and quality. Journal of Clinical Nursing, 25(1–2), 163–174. https://doi.org/10.1111/jocn.13113.

VanderKooi, M. E., Conrad, D. M., & Spoelstra, S. L. (2018). An enhanced actualized DNP model to improve DNP project placements, rigor, and completion. Nursing Education Perspectives, 39(5), 299–301. https://doi.org/10.1097/01.NEP.0000000000000384.

Waldrop, J., Caruso, D., Fuchs, M. A., & Hypes, K. (2014). EC as PIE: Five criteria for executing a successful DNP final project. Journal of Professional Nursing, 30(4), 300–306. https://doi.org/10.1016/j.profnurs.2014.01.

Weiskopf, N. J., & Weng, C. (2013). Methods and dimensions of electronic health record data quality assessment: Enabling reuse for clinical research. Journal of the American Medical Informatics Association, 20(1), 144–151. https://doi.org/10.1136/amiajnl- 2011-000681 January.

L.L. Costa, et al. Journal of Professional Nursing 36 (2020) 666–672

672