A Critical Review of Transition-to-Professional-Practice Programs:
Applying a Standard Model of Evaluation
Abstract
Introduction: Although transition-to-professional-practice programs (TPPPs) for newly graduated
registered nurses are considered beneficial, no consensus exists about their ideal components.
Thus, evaluating the effectiveness of TPPPs and identifying the components that strongly
influence outcomes is important. Aim: The aim of this critical review was to demonstrate the
usefulness of applying the Kirkpatrick model to evaluate the effectiveness of TPPPs. Methods: A
review of the literature between 2010 and 2016 was conducted, and TPPPs were assessed using
the Kirkpatrick model for evaluating educational outcomes. Additionally, all selected studies were
analyzed for risk of bias in reporting results or inferences claimed as a consequence of new
registered nurse participation in a TPPP. Risk of bias was designated as high, medium, or low.
Results: The database search identified 86 studies, and 26 met the inclusion criteria for critical
review. Based on their analysis, the authors found that TPPPs are often evaluated by their
respective organizations, not by an independent body. Therefore, most study findings reported that
TPPPs achieve their aims, but the analysis revealed a high risk of bias in those findings.
Conclusion: The Kirkpatrick model revealed a high risk of bias in the reporting of TPPP
outcomes, indicating a need for evaluations based on rigorous research.
Despite the significant need for new graduate registered nurses (RNs) (World Health
Organization, 2017; RCN Labour Market Review, 2016), the advanced beginner status of new
graduates raises concerns about patient safety (Clarke & Donaldson, 2008). Employers may
perceive new graduate RNs as unprepared to provide safe, effective patient care (Spector & Li,
2007). Furthermore, the clinical environment is unpredictable, and new RNs can experience reality
shock as well as a high level of anxiety about making errors in clinical practice (Mellor & Greenhill
2014; Kramer 1974). Stress and anxiety alone can contribute to poor decision making (Benner,
2004; Lea & Cruickshank, 2007; Spector, 2011; Spivak, Smith & Logsdon, 2011).
Transition-to-professional-practice programs (TPPPs) can help new RNs and the organizations
they serve. However, no consensus regarding the ideal components or interventions of TPPPs
exists (National Council of State Boards of Nursing [NCSBN], 2013; Rush, Adamack, Lilly, &
Janke, 2013; Strauss, Ovnat, Gonen, Lev-Ari, & Mizrahi, 2016; Tuckett, Eley & Ng, 2017). Some
TPPPs provide educational activities and no other supports (Kaddoura, 2010). Others are
intricately designed with elaborate networks of tangible and intangible supports for new nurses
(Kawolski & Cross, 2010; Kramer, Lindgren, High, Ocon, & Sanchez, 2012). Therefore,
evaluating the effectiveness of TPPPs and identifying the interventions that have a stronger
influence on outcomes is critical.
In 2011, the NCSBN responded to documented concerns about patient safety and lobbied for
completion of a TPPP as a requirement of re-licensure after the first year of practice (Bratt &
Felzer, 2011). NCSBN (2013) collaborated with 35 organisations to develop a transition-to-
practice (TTP) model. Then, a multisite, randomized comparison of existing TPPPs with the
NCSBN TTP model was undertaken. Although the results were insufficient to promote the NCSBN
requirement, they provided evidence that high-quality, evidence-based TPPPs have a beneficial
effect on new RN safety outcomes (NCSBN, 2013).
Critical Review of Programs
For many years, TPPPs were used without routine impartial evaluation (Rush, Adamack, Lilly, & Janke,
2013; Parker et al., 2014; Edwards, Hawker, Carrier, & Rees, 2015). Today, with resources declining
internationally, ensuring learning effectiveness is essential (Morgan & Astolfi, 2015). Thus, the aim of
this critical review was to demonstrate the effectiveness of using a standard model of evaluation, the
Kirkpatrick model, to evaluate TPPP interventions and program outcomes.
Literature Search
An initial search was conducted using the term “graduate nurse program.” Additional keywords
plus subject headings were identified from the titles and abstracts of the retrieved articles. Next,
a formal search of the academic literature was conducted using all permutations of the identified
synonyms for “graduate,” “nurs*,” and “program.”
CINAHL, Science Direct, Google Scholar, Medline, Science Citation index, Social Science index,
Informit, and Proquest were searched for articles published between 2001 to 2016. Keywords were
as follows: graduate OR neophyte OR "newly registered" OR novice OR transition OR "transition
to practice" OR TTP AND nurs* AND [program OR residency OR mentorship OR preceptorship].
The search yielded 43 full-text articles, which were assessed for eligibility (n = 43); however, the
authors decided to provide a more contemporary view by excluding articles from before 2010 (N =
26). See Figure 1.
Figure 1. PRISMA Search Strategy (Moher et al 2009)
Studies were included if the participants were new graduate RNs. Articles were reviewed using the
title only and retrieved if they were descriptive accounts or evaluative accounts or if they reported
the lived experience of graduates in any aspect of a program. These articles were considered
most likely to yield sufficient detail about the elements of the graduate program to contribute to the
review. Articles were included if they were primary research reports of programs aiming to assist
transition from undergraduate nursing education to professional nursing practice (See Table 1).
Table 1. Inclusion-Exclusion Criteria for Literature Search
Critical Review of Programs
For many years, TPPPs were used without routine impartial evaluation (Rush, Adamack, Lilly, &
Janke, 2013; Parker et al., 2014; Edwards, Hawker, Carrier, & Rees, 2015). Today, with
resources declining internationally, ensuring learning effectiveness is essential (Morgan & Astolfi,
2015). Thus, the aim of this critical review was to use a standard model of evaluation, the
Kirkpatrick model, to evaluate TPPP interventions and program outcomes.
Evaluation of Programs
Program characteristics, including the location of the TPPP, duration, follow-up, and educational
strategies, were recorded to provide context. In accordance with the aims of this review,
analyses for both educational outcomes and risk of bias were undertaken to determine the rigor
of the individual studies and the reliability of the evaluations.
Educational Outcomes
For more than 40 years, the internationally recognized Kirkpatrick Model of Evaluation (Kirkpatrick
& Kirkpatrick, 2006) has been used to evaluate educational and training programs (Galloway, 2005).
The model allowed for reported outcomes of each TPPP to be aggregated and analyzed according to
four progressive levels of evaluation. Regarded as an appropriate model for transition programs, the
Kirkpatrick model was recently used in the construction and implementation of a comprehensive
evaluation plan for the Wisconsin Nurse Residency Program (Meyer Bratt, 2013
The four levels of evaluation are as follows (Table 2):
1. Reaction: Studies measure new graduate RNs’ views of the learning
experience, 76
organization, presentation, content, teaching methods, and quality of instruction.
2. Learning: Studies consider the extent of learning. Level 2A refers to learning measured
with regard to changes in the new graduate RN’s attitude, and level 2B refers to learning
measured with regard to the new graduate RN’s knowledge and skills.
3. Behavior: Studies report changes in the new graduate RN’s behavior resulting from
the TPPP.
4. Results: Results are reported as changes in organizational climate (level 4A)
or improvement in the new graduate RN’s performance (level 4B).
Some consider the four-level model too simplistic (Bates, 2004). Also, concern exists regarding
the subjective nature of levels 1 and 2 as well as claims that the model does not include
contextual influences when assessing the transfer of behavior or skills to the workplace, such as
the learning culture of the organization (Bates, 2004; Galloway, 2005). However, the revised
Kirkpatrick model (2006, p. 24) that was released after these critiques describes five different
types of organizational climate or contextual influences on learning outcomes: preventing,
discouraging, neutral, encouraging, and requiring. After careful consideration of the evidence and
commentary available, the authors believed that the revised Kirkpatrick model is an appropriate
contemporary tool for analysis.
Risk of Bias
Because of the high level of self-reporting by TPPP providers, potential bias in evaluations must
be considered. Risk of bias in this article refers to any systematic error in the results or inferences
claimed as a result of new RN participation in a TPPP (Higgins & Green, 2011). The assessment
of risk of bias has been designated as high, medium, or low (Table 3). A high-bias study typically
presented self-reported results with minimal evidence of objective inquiry or had another aspect
that affected the rigor or results. Conversely, a low-bias study showed more evidence of rigor, and
the reported findings were considered more accurate. These distinctions are important if future
TPPPs are to be developed based on the best available evidence.
Consequently, each study was assessed for risk of bias to determine the quality of the research
against recognized criteria for both qualitative and quantitative studies. The risk of bias in qualitative
studies was assessed using Law et al.’s critical review form for qualitative studies (1998a). Criteria
included study purpose, literature relevance, study design, theoretical perspective, methods used,
sampling, data collection integrity, procedural rigor, data analysis, auditability, theoretical
connections, overall rigor, conclusions, and implications. Quantitative studies were analyzed using
Law et al.’s critical review form, for quantitative studies (1998b). Criteria included study purpose,
literature, design, sample, outcomes, analysis and results, conclusions, and implications. For both the
Kirkpatrick evaluation and the assessment of bias, the authors undertook initial analysis
independently; when differences arose, the authors resolved them through consultation and
consensus.
Results
A total of 86 studies were identified from the database search, and 26 articles met the inclusion
criteria. Studies were primarily conducted in the United States (n = 18). Two were conducted in
Australia, and two in the United Kingdom. One study was conducted in each of the following:
Brazil, Canada, Saudi Arabia, and United Arab Emirates. Extensive variation was noted regarding
the specialties in which TPPPs occurred. The majority were in acute care environments, such as
surgical, medical, emergency, and critical care, the operating room, and the recovery room
(Beyea, Slattery & von Reyn, 2010). The TPPPs were in large metropolitan hospitals or small rural
hospitals (Bratt & Felzer, 2011).
Of the 26 studies, 8 evaluated their TPPPs on all levels of Kirkpatrick’s model, and 23 reported
changes in new RNs’ behavior (level 3). The results of training (level 4) are also frequently
reported, with 23 studies analyzing change in the organizational climate (4A) and 23 studies
considering new RN outcomes (4B). Attitude and level of confidence (level 2A) was assessed by
15 studies, and knowledge and skill acquisition (level 2B) was assessed by 21 studies. This
acquisition included elements of professional knowledge development, clinical competence, and
critical thinking. The reaction of participants to the training (level 1) was considered by 17
studies. Three studies had a control group.
Change of behavior was reported in 23 studies and related to observed professional socialization,
professional competency, confidence with nursing skills, socialization, and willingness as part of
the team. Of the evaluated studies, 21 (83%) reported a positive result from their educational
interventions, and 5 (14%) achieved mixed or variable results in achieving desired goals. In 12
studies (43%), follow-up did not occur. In studies with follow-up, the time varied from 6 months to
5 years. In 4 studies (15%), follow-up was 6 months to 9 months; in 6 studies (23%), follow-up
was 1 year. Individual programs had follow-up at 18 weeks, 4 months, 2 years, and 5 years. Most
follow-up focused on retention in the workforce and factors affecting attrition.
Study design provided context to the evaluations. High risk of bias was found in 10 studies;
medium risk of bias was found in 6 studies; and low risk of bias was found in 10 studies. Those
studies with a high risk of bias reported outcomes of their programs as being very successful,
were non research based, and were more closely aligned with continuous improvement.
Conversely, in studies with a lower risk of bias, a research method reflecting a higher level of
confidence had been adopted, and follow-up was more likely. Thus, the outcomes that were
reported in the low bias studies were considered more rigorous and higher quality.
The Kirkpatrick model enabled an effective and rigorous comparison of research on TPPPs.
During the analysis, the authors saw that TPPPs were diverse and that developing one program to
fit all situations was unlikely. Examined through the broad lens of the Kirkpatrick model, nurse
educators in the professional clinical environment can use this evaluation of TPPP studies to
inform program development.
Program Design
Some studies were focused on employment and retention of new RNs as an impetus for
recruitment (Meyer-Bratt, 2013). Although inadequate staffing levels risk patient safety, the risks
of employing an advanced beginner RN need to be acknowledged when goal setting. Ebright
(2010) posed the following question about the employment of the new graduate RN: “What
distribution of care delivery or alternative assignment of RNs would assure that all new graduate
nurses would receive mentoring for their assigned patient care and work management
responsibilities from an experienced RN every shift?” For an organization that makes patient
safety a goal for a TPPP, such questions would be measured as an outcome.
After TPPP goals are established, each element of the evaluation process must be monitored,
and a measurable evidence-based approach must be applied. This approach determines whether
a program is succeeding and provides evidence on which interventions are not working. For
example, leadership from the American Nurses Credentialing Center (2016), which offers the
Practice Transition Accreditation Program, and the Versant New Graduate RN ResidencyTM
program have made progress toward evidence-based practice for transitioning nurses. Although
both have used control groups when evaluating their programs, no research has focused on
individual interventions and the relationship between patient outcomes and participation in a
nursing residency program (Letourneau, & Fater, 2015. In 1999, the Versant New Graduate RN
Residency program was piloted in a pediatric hospital while subjected to systematic evaluation
using a control group (Ulrich et al., 2010). When compared with the control group, the intervention
group demonstrated better outcomes (Ulrich et al., 2010). The program has since been extended
to other paediatric and adult acute care facilities and is now offered to other health care facilities in
the United States (Ulrich et al., 2010).
Although the Versant model was reported as successful, elements of it could be further
investigated, particularly from the perspective that the residency program is offered as a package
of interventions, making it difficult to discern which interventions contribute most to the program
outcomes (Rush et al., 2013). For example, Figueroa, Bulos, Forges, & Judkins-Cohn (2013) in
the background to their research asserted that the effect of the Married State Preceptorship Model
on the perceptions of new RNs and preceptors had not fully been addressed in the academic
literature. Furthermore, Spector (2015, p. 1) in the NSCBN Transition-to-Practice Study found that
new RN competency improved over the course of a TPPP whether or not the new RNs had
additional didactic content. These observations emphasize the need for consideration of both
program design and systematic evaluation of interventions rather than design alone.
The Kirkpatrick model (Kirkpatrick & Kirkpatrick, 2006) provides such a measure of outcome.
Based on Kirkpatrick’s evaluation model, the authors’ study found a diverse scope and extent of
TPPP evaluations. Broadly, the studies reviewed suggest that, at least in the short term, TPPPs
generate new RN satisfaction, extend learning, positively change behaviour, and result in
improved new RN performance and improved organizational climate (Ulrich et al., 2010; Spector
& Echternacht, 2010). However, a tendency toward bias exists in many of these results. Overall,
further rigorous, detailed evaluation of the impact and effectiveness of TPPPs and their adopted
interventions are needed (Rush et al., 2013; Tuckett, Eley & Ng, 2017).
Table 3. Outline of studies 2010 - 2016
Implementation and Delivery
TPPPs are often implemented in response to perceived staff shortages or financial benefits. Thus,
tension often exists between the needs of the new RN and the goals of the organization (Cadmus,
Salmond, Hassler, Black, & Bohnarczyk, 2016). An evaluation of TPPPs can provide hindsight for
nurse educators by revealing retrospective information about learning, experiences, behavior
changes, and results. Such analysis allows organizations to use limited resources where they are
most effective and to avoid diluting components of programs without realizing their significance,
such as adopting a low-support preceptorship model where the preceptor is available for questions
but does not share the workload with the new RN (Spector, 2015; Cadmus et al., 2016;
Figueroa et al., 2013).
Monitoring, Evaluating, and Reporting
Nurse educators are not required to report evaluations of TPPPs in the academic literature.
Yet without doing so, future TPPPs are compromised by the lack of rigorous information that
informs their development.
In the early development phase, TPPPs are often evaluated by their organizations, rather than
by an independent body, and accounts are rarely published. Health services are therefore
encouraged to collaborate with universities to establish and maintain a research focus so
critical evaluation of all TPPP interventions can take place. Although findings were mostly
recorded as positive in this study, the risk of bias is often high and the follow-up is variable.
Organizations indicate that their TPPPs are effective in achieving a successful transition;
however, many confounding factors can influence the evaluation. The recruitment process
may be rigorous to ensure the best possible RN candidates and hence safeguard the
program’s success. As Patterson et al. wrote:
Criteria for application included a cumulative grade point average of 3.0 or higher;
positive recommendations from a nurse manager and two nursing faculty members; an
interview with a . . . clinical educator, nurse manager, and emergency staff nurse; a 200-
word essay describing the applicant’s reasons for choosing emergency nursing; evidence
of nursing leadership activities in school; and previous related health care work
experience. (2010, p. 204)
Other potential built-in safeguards may include the requirement that participants sign
employment contracts for periods of up to 2 years, which ensure a high level of success when
evaluation is based on retention. For example, in one study, “All nurses participating in the
residency program were required to sign a 2-year employment contract, which included the
12-month program” (Holland & Modderman, 2012, p.332).
Moreover, more TPPPs must use control groups. If control groups are not used for comparison,
determining if time and experience alone are responsible for new RN improvement is difficult.
Attributes such as competency, skills, attitude, and self-confidence can all show improved
results over time without any intervention. Thus, to best assess the value of TPPPs and
strengthen the evidence, educators must compare new RNs that participated in a TPPP
(experimental group) with those that did not (control group).
Several models have used control groups. As noted, NCSBN completed a controlled study
comparing their TTP with existing TPPPs. Also, the Versant New Graduate RN Residency program
has been evaluated using a control group (Ulrich et al., 2010). The initial pilot study had an
intervention group that demonstrated better outcomes than a comparison group (Ulrich et al.,
2010), and the program was reported to be successful in other pediatric and adult acute care
facilities. Because of these controlled studies, the NSCBN TTP model and Versant program are
offered nationally in the United States.
Use of the Kirkpatrick Model
Further rigorous evaluation is the key to improving TPPP research. Kirkpatrick’s model
(Kirkpatrick
& Kirkpatrick, 2006) provides a tool that can be applied across an assortment of programs.
Thus, the authors advocate its use as an ongoing evaluation program.
To evaluate various programs in a range of settings requires further information about the
reaction (level 1) of participants to the learning experience (favorable or unfavorable). The lack
of this information in the reviewed literature compromised the authors’ ability to assign “level
achieved” scores. According to Kirkpatrick & Kirkpatrick (2006, p. 27), participants need to react
favorably “otherwise they will not be motivated to learn.” In some studies, determining changes
in attitude was difficult. Attitude and level of confidence (level 2A) and knowledge and skill
acquisition (level 2B) were evaluated (Ostini & Bonner, 2012). Some studies stated that new
RNs were assessed, but the studies did not provide outcomes (Fielden, 2012). Other studies
were unclear or anecdotally reported outcomes without supporting data (Silva, Cordeiro,
Fernandes, da Silva, & Teixeira, 2014). Changes in new RN’s behavior (level 3), change in the
organizational climate (level 4A), and improvement in the new graduate RN’s performance
(level 4B) were more consistently reported, although clarity was occasionally lacking (Silva et
al., 2014; Banks et al., 2011; Holland & Modderman, 2012)
Based on the Kirkpatrick evaluation model, this study found a diverse scope and extent of TPPP
evaluations. Broadly, the studies reviewed suggest that, at least in the short term, TPPPs generate
new RN satisfaction, extend learning, positively change behaviour, and result in improved new RN
performance and improved organizational climate. However, the tendency toward bias in the
reviewed studies suggests the need for thorough consideration of rigor when interpreting results.
Limitations
Identifying all TPPP evaluations during the period from 2010 to 2016 was challenging because
many are not reported in the academic literature. Some TTTPs were designed uniquely for
organizations, and titles were not transparent. Examples of this diversity include “The Bridge
Program,” “Nurses Nurturing Nurse’s Mentorship program,” “Married State Preceptorship
model,” and “Flying Start NHS.”
Researchers had to rely on the written word for the content, intent, evaluation, and risk of bias in
each reviewed program. Evaluating effectiveness requires that articles have quality descriptions of
the interventions because “without a complete published description of the intervention, other
researchers cannot replicate or build on research findings” (Hoffman et al., 2014). However, the
authors were flexible, and a degree of latitude was applied when evaluating the various
programs. The results may be influenced by the authors’ interpretation.
Conclusion
The diversity and variable structure of TPPPs requires a standardized means of evaluation, in the
form of the Kirkpatrick model, to make learning more widespread and applicable. Successful
evaluation of the four levels requires advanced research expertise and partnerships among health
services and the tertiary sector. This approach determines whether a program is succeeding and
provides evidence on which interventions are not working. Effective reporting of the design,
implementation, and outcomes of individual TPPPs in the literature will significantly enhance
accountability, scholarship, transferability, and confident execution in other applicable contexts.
References
American Association of Colleges of Nursing. (2015). Nursing shortage fact sheet. Retrieved
from http://www.aacn.nche.edu/media- relations/NrsgShortageFS.pdf
American Nurses Credentialing Center. (2016). Practice transition accreditation
program. Retrieved from http://nursecredentialing.org/PTAP-AccreditationManual
Banks, P., Roxburgh, M., Kane, H., Lauder, W., Jones, M., Kydd, A., & Atkinson, J. (2011).
Flying Start NHS(TM): Easing the transition from student to registered health professional.
Journal of Clinical Nursing, 20(23-24), 3567–3576. Retrieved from
http://dx.doi.org/10.1111/j.1365-2702.2011.03796.x
Bates, R. (2004) A critical analysis of evaluation practice: The Kirkpatrick model and the
principle of beneficence. Evaluation of Program Planning, 27, 341–347.
Benner, P. (2004). Using the Dreyfus Model of Skill Acquisition to describe and interpret skill
acquisition and clinical judgment in nursing practice and education. Bulletin of Science,
Technology
& Society, 24(3), 188-199. doi:10.1177/0270467604265061
Beyea, S. C., Slattery, M. J., & von Reyn, L. J. (2010). Outcomes of a simulation-based
nurse residency program. Clinical Simulation in Nursing, 6(5), e169-e175. Retreived from
http://dx.doi.org/10.1016/j.ecns.2010.01.005
Bratt M., Felzer, H. (2011). Perceptions of professional practice and work environment of new
graduates in a nurse residency program. Journal of Continuing Education in Nursing, 42(12),
559– 568. doi:10.3928/00220124-20110516-03
