method

IDE 641 SPRING 2019

Eagle Productivity Solutions Creative & Interactive Team Onboarding

EValuation Plan 2019

Prepared by: Sarah Bernhard Zachary Schuster Abdullah Albelehy

Table of Contents

Introduction and Background 02

Evaluation Purpose 03

Evaluation Audience 04

Questions 04

Decisions 06

Methods 07

Sample 07

Instrumentation 08

Limitations 09

Logisitics and Timelines 10

Budget 11

Introduction and Background

Eagle Productivity Solutions (or just “Eagle”) is a 200-person training vendor based in Rochester, New York. Since its establishment in 1988, Eagle has partnered with local and international clients to design, develop, and deliver custom training solutions, mostly focused on proper technology usage. Along with traditional offerings like instructor-led training and printed reference materials, Eagle also offers more modern solutions like web-based reference guides and on-demand video tutorials. Although most of its customers are pharmaceutical companies – Pfizer, Bayer, and Bristol-Myers Squibb, to name three – Eagle’s client base has expended in recent years to include non-pharma clients such as McDonald’s, Facebook, and the City of Rochester.

The forthcoming evaluation plan concerns one of Eagle’s critical production teams, Creative & Interactive (C&I). The C&I team, itself, is one of several teams that comprise the Solutions Development department, which is responsible for producing the deliverables sold to Eagle’s clients as part of a scope of work. C&I, specifically, is responsible for tasks such as the graphical enhancement of deliverables like presentations and reference cards, the assembly and production of eLearning courses, and the programming of custom websites. The personnel who complete these tasks are officially called Creative Developers or Creative FX Developers, but (depending on their specific area of expertise) they are comparable to graphic designers, video producers and editors, and web developers.

Despite Eagle’s reputation as a leader in custom training solutions, most of its employees have likely heard (or even said) some variation of the following phrase: “Eagle can train anyone, except its own employees.” Solutions Development, specifically, has been criticized for its lack of formal onboarding* for its new hires. Oftentimes, seasoned developers can be heard telling newcomers that most people learn through “trial by fire,” not the training provided by management. The general perception is that the lack of formal onboarding has resulted in low-quality products, inconsistent quality standards, and job dissatisfaction, among other side effects. This lack of onboarding inspired the forthcoming evaluation plan when it was brought to one of the evaluator’s attention by Stephanie Giles, the Director of Solutions Development. Giles expressed the need to begin evaluating the current onboarding program in response to the aforementioned issues, especially as Eagle continues to take on more clients, expand its offerings, and cultivate its own talent.

Although an all-inclusive evaluation of the Solutions Development department at large may be planned for the future, the one outlined in this document concerns the C&I onboarding process specifically, per Giles’ recommendation to use it as a starting point. Currently, their onboarding process is rarely consistent across new hires, but it usually entails a face-to-face overview of general Solutions Development department processes, then additional face-to-face training and accompanying reference resources dependent on their role. It should be noted that, due this varied nature of the onboarding process, a comprehensive description is difficult to achieve in the planning phase. Although not an explicit goal of this evaluation, it can be anticipated that in investigating the current process further, a better, more thorough understanding of it can be established.

*Important Note: For the purposes of this evaluation, the term “onboarding” refers to the orientation activities administered by Eagle to help acclimate new employees to the responsibilities and expectations of their role.

Purpose

The motivation behind this evaluation is to calculate the effectiveness of Eagle Productivity Solutions’ Solutions Development’s current onboarding process. Eagle Productivity Solutions was recently named one of Rochester, NY’s Top 100 companies and was included in INC Magazine’s list of the fastest growing companies in the United States. For the purpose of this evaluation, the practitioners have be asked to focus on the Creative and Interactive team within the Solutions Department. Conducting a robust evaluation to access the overall effectiveness of the onboarding program could not come at a more critical time. Due to intensified market demands and increased workload, Eagle Productivity Solutions expects the Creative and Interactive team to grow in the upcoming months. It has also been difficult to retain workers on the Creative and Interactive team as the turnover rate is high. Bringing in new hires has proved to be extremely costly in time and financial resources. No previous evaluations have been conducted on the current onboarding process.

This report will serve as a summative and formative evaluation of Eagle Productivity Solution’s Creative and Interactive’s onboarding process. The organization’s goals are to determine if the current onboarding program has been successful and make recommendations to improve their process in the future. In addition to evaluating the effectiveness Eagle Productivity Solution’s Creative and Interactive’s onboarding process, this evaluation also identifies specific sub-purposes.

1. Collect information concerning the overall impact of the onboarding process on Creative and Interactive team members

2. Collect new hires’ confidence levels upon completing their onboarding

3. Collect attitudes and opinions on the current onboarding process.

4. Collect the average percentage of rework needed for deliverables completed by new hires

5. Collect the average time elapsed between new hires’ start dates and when they are considered competent enough to operate on their own (i.e., without required pre-review or frequent intervention from a manager/supervisor

6. Collect information regarding where progress can be made in the Eagle Productivity Solutions onboarding process.

7. Collect information to analyze if the onboarding program achieved its original goals and intentions.

Evaluation Audience

Audience for the Evaluation Report

Immediate audience of this evaluation report include:

The primary audience for this evaluation

• Stephanie Giles - Director of Solutions Development (main client/sponsor)

• Andrew McGinnis - Creative & Interactive Supervisor.

Primary stakeholders include the people most directly involved in or affected by the evaluation.

The secondary audience for this evaluation includes

• Senior/Upper Management - General Manager (Alex Orlando), VP of Production Services (Sharon Sienkiewicz)

• Human Resources - HR Manger (Michele Tucker)

• Creative & Interactive Team Members - Creative Developers, Web Programmers, Creative FX Developers, Studio Leads, etc.

Secondary audiences are any people judged to have a stake in the evaluation and thus a right to know about its methods and results

Questions

The evaluation will seek to answer the following primary and secondary questions (note: the process through which these questions were formulated is described following the questions, themselves):

· Primary Question #1: Is the current onboarding process for Eagle’s Creative & Interactive employees effective?

· Do developers feel confident in their ability to perform their duties upon completion of formal onboarding?

· Is the onboarding process more effective for some roles than it is for others?

· How well to new hires retain the information and skills they learn as part of their onboarding?

· Primary Question #2: What key skills do Creative & Interactive employees learn over the course of their onboarding?

· What skills are considered critical for each role?

· What skills are developers expected to already have when hired?

· How are new skills and concepts conveyed during onboarding?

· How is mastery of these skills and concepts measured or validated?

· How soon do developers begin to apply these skills after their hire date?

Since its inception, the parameters of this evaluation have evolved for a multitude of reasons. One of the biggest decisions was to reduce the overall scope of the evaluation, narrowing the focus from the entire Solutions Development department to just the Creative & Interactive team. The evaluators first identified the need to narrow the scope early in the project, as evaluating a highly-complex and varied onboarding program for a department of over fifty people would have proved cumbersome for a three-person team with limited time and resources. Once this was established, the primary stakeholder was asked which sub-department or team within Solutions Development would benefit most from the proposed evaluation. She stated that, due to its relative newness and high potential for growth, the Creative & Interactive team would be the best candidate. Given its size (roughly 20 people), the evaluators agreed this would be more manageable and decided to proceed with the narrowed focus on just Creative & Interactive.

Another key decision was to clarify what, exactly, is meant by “onboarding.” Although it is a common term for those familiar with corporate culture and procedures, those who have never worked in an office environment may not immediately recognize the term. Likewise, the term “onboarding” may have a variety of meanings, even for those who are familiar with it based on their own experience. Given its significance in this evaluation, the evaluators determined it was necessary to clearly establish the term’s meaning and assumptions before proceeding further, resulting in the definition cited in the “Introduction and Background” section of this document.

A number of critical decisions were also made while formulating the evaluation’s primary and secondary questions. For example, the original proposal included several primary and secondary questions that have since been eliminated or reprioritized. More specifically:

· The question “What percentage of billable work completed by onboarded new hires needs to be reworked by more experienced team members?”, originally a secondary question of Primary Question #1, was removed. To answer this question, the evaluators considered analyzing timecard and project data from Salesforce, the system Eagle uses to track its project metrics. While answering this question would have provided quantifiable evidence of the impact and efficacy of the current onboarding program, the evaluators anticipated such data would be too complex and voluminous to analyze fully in the given timeframe. The proprietary nature of such data could have also made it difficult to obtain, despite one of the evaluators being an Eagle employee. Additionally, the evaluators decided this data may not be entirely reliable, as it may have been influenced by a variety of factors unrelated to the efficacy of the responsible team members’ onboarding (e.g., client demands, timeline shifts, system readiness for technical projects, etc.).

· The question “Is the onboarding process more effective for some roles than it is for others?” was originally a primary question, but was changed to a secondary question of the current Primary Question #1. Although there are fewer unique roles within the Creative & Interactive team than there are in the department at large, a degree of variety does still exist, meaning this question is still relevant. As confirmed by the primary stakeholder, even within Creative & Interactive, different roles have different onboarding experiences. Therefore, it can be assumed that a new hire’s role may have a direct impact on how effectively they adapt to their position, which directly relates to the topics addressed by Primary Question #1, making it an ideal sub-question.

· The original Primary Question #2, “What key skills do developers learn while onboarding?” was revised slightly to accommodate the narrowing of the overall scope, as well as to be more specific. This original wording also revealed the previously-described need to more clearly define the term “onboarding” at forefront of all documentation, as doing so ensures all instances of the term can be easily understood by all audiences.

· The original proposal also included the primary question “Who is responsible for new hires’ onboarding?”, but it was eliminated along with its secondary questions. Although the evaluators agree that the personnel conducting the onboarding certainly play an important role in whether or not the process is effective, the subject was deemed too broad and complex to include with the given timeframe.

In addition to the decisions regarding the evaluation’s scope and driving questions, it is worth noting that all the evaluators’ decisions were almost universally influenced by two of the greatest limitations: distance and access to research subjects. These factors are described in greater detail in the “Limitations” section of this document, but generally speaking, both had a substantial impact on the decision-making process. Because the evaluation site is in Rochester instead of Syracuse, the evaluation is structured to accommodate two-thirds of the team’s inability to be physically at the evaluation site. As a result, certain methods of data collection have been chosen specifically because they can be administered virtually – for example, surveys, questionnaires, and remote interviews with relevant subjects. Similarly, the client expressed the need to minimize intrusion on subjects and interviewees’ working time, ruling out more time-consuming activities such as focus groups or extended interviews.

Decisions

In consulting Giles, it has become apparent the evaluation results will be welcomed by Eagle management for their potential to help enhance C&I's onboarding process. Although it is difficult to predict the exact decisions this evaluation will influence, the following subjects are likely to be addressed following dissemination of the report:

1. The amount of time and resources dedicated to onboarding a single new hire. If the evaluation results indicate that not enough time and resources are dedicated to onboarding C&I team members, Eagle management may decide to increase them to ensure a more efficient and worthwhile experience.

2. Creation and deployment of a uniform onboarding process for C&I. If the evaluation results indicate that the process's inconsistency is a major detractor from its efficacy, Eagle management may decide action is required to create a more standardized curriculum for all new team members.

3. Creation and deployment of "add-on" onboarding resources, dependent on role. If the evaluation results indicate existent, role-specific resources are not currently sufficient, Eagle management may decide action is required to create a more comprehensive, role-specific curriculum to accompany the general C&I onboarding described in bullet 3.

4. Specific modalities and methods to be used for onboarding. Depending on responses received from interviews and surveys, the evaluation results may help Eagle management decide what modalities and methods could be most effective for onboarding - for example, instructor-led training, on-demand videos, etc.

Look at the section for “methods” on the attached example. You really need to explain each method to the reader. You really need to explain each method to the reader.

Also, hash out the sampling more. Who will be sampled for each section? How many/who will be interviewed? How many/who will be sent the satisfaction survey?

Sample

Sample for the Evaluation Report

Stephanie Giles, Andrew McGinnis, Roger Trinh, Christine Wolfanger, Kama Post, various other members of the Creative & Interactive Team (for surveys/questionnaires, around people from varying roles)

Methods

• Surveys/questionnaires (distributed to C&I team members)

• Interviews (conducted with both stakeholders and C&I team members)

• Literature review (analysis of existing materials and documentation)

• Anecdotal record form (ask people to give a brief description or notable event in their onboarding experience).

Instrumentation

This evaluation will measure the Eagle Productivity Solutions onboarding process’s effectiveness by using the Kirkpatrick Four-Level Training Evaluation Model. The Kirkpatrick Four-Level Training Evaluation Model is a tool designed to evaluate training programs through a sequence of stages. At each stage of the evaluation, our practitioners will utilize different instrumentation to identify the success of the onboarding process, areas of improvement and other measurables. A description of the Kirkpatrick Four-Level Training Evaluation Model and each instrument used in the evaluation is located below.

Kirkpatrick Four-Level Training Evaluation Model

· Level 1 – Reaction: End-users will be sent an Eagle Productivity Solutions onboarding satisfaction survey to gauge its effectiveness, what they learned, how they value the training, areas to improve, etc.

· Level 2 – Learning: Creative and Interactive Developers will be interviewed to access their confidence in understanding their specific job duties, corporate quality standards, etc. after the onboarding process.

· Level 3- Behavior: Due to the limited amount of time and scope of our evaluation, the evaluation team will not conduct a formal evaluation on level 3 of the Kirkpatrick Four-Level Training Evaluation Model.

· Level 4 – Results: At this stage the evaluation will clearly state its final results. Outcomes will be based on a combination of the interviews, user satisfaction survey and behavioral changes after training.

Instrument 1 - Eagle Productivity Solutions Onboarding Satisfaction Survey

Instrument Description

To determine new and existing employee satisfaction with Eagle Productivity Solutions onboarding process, the practitioners of this evaluation will utilize a satisfaction survey. The Eagle Productivity Solutions Onboarding Satisfaction Survey will be comprised of 15 questions using a Likert-type scale to measure trainee’s attitudes, feelings, opinions, etc. with the current onboarding methods and its efficiency. Trainee’s will score their satisfaction of the current onboarding process using a 5 point scale and will also have the option to include personal remarks about each question with optional comments box.

1=Strongly Disagree, 2=Disagree, 3=Neutral, 4=Agree, 5=Strongly Agree

The Eagle Productivity Solutions Onboarding Satisfaction Survey will include questions regarding the user’s ability to perform their required duties following the completion of the onboarding process, skills learned during the trainings, important areas that were not covered during onboarding, etc.

Instrument Type

For this survey, the evaluators will be utilizing the Qualtrics Survey Tool and Research Suite. The tool is able to administer an online anonymous survey. Creative and Interactive team members will self-report their satisfaction using the 5-point Likert-type scale and make additional comments in fillable text boxes.

Instrument Audience

The principal audience that will be surveyed is the Creative and Interactive Team’s developers, managers and supervisors.

Objectives of the Instrument

End-users will be sent an Eagle Productivity Solutions onboarding satisfaction survey to gauge its effectiveness, what they learned, how they value the training, areas to improve, etc.

Instrument 2 - Eagle Productivity Solutions Onboarding Learning and Behavior Interviews

Instrument Description

Trainee’s and managers of the Eagle Productivity Solutions onboarding process will be interviewed to access their confidence in understanding their specific job duties, corporate quality standards, etc. after the onboarding process. Our team of independent evaluators will gather information from the Creative and Interactive Team’s developers, managers and supervisors through a semi-structured interview approach. A variety of interview questions will be used to access each team member’s satisfaction, discontent, confidence and other feelings related to the Eagle Productivity Solutions onboarding process. Example questions include: “Do developers feel confident in their ability to perform their duties upon completion of formal onboarding?” and “What key skills do developers learn while onboarding?” The interviews with be a hybrid of in-person/virtual meetings with one evaluator on site, and two evaluators attending through video conferencing. Do to the interview participants busy schedules, the evaluators will only spend 15 minutes with each team member. Using video conferencing technology, the evaluators will capture and record each interview for future use and assessment.

Instrument Type

A set of formal and informal procedures will be used to access satisfaction and the success of the Eagle Productivity Solutions onboarding process. Evaluators will use a semi-structured interview approach utilizing common job principles and set questions regarding their scope of work. This will allow the evaluator to adapt questions to individuals, including open and closed questions, scripts, etc. throughout the session to make the best use of time.

Instrument Audience

The principal audience that will be interviewed is the Creative and Interactive Team’s developers, managers and supervisors.

Limitations

As in any evaluation, the evaluators are faced with a number of limitations. The most significant of these limitations are listed and described below and referenced in other sections of this document, where appropriate:

· Time: The evaluation itself is expected to be executed between March 3rd, 2019, and April 28th, 2019, a total of 56 days or approximately eight weeks. This relatively short amount of time dictated many of the decisions with regard to the evaluation’s scope and focal points, as described in the “Decisions” section of this document.

· Distance: The primary evaluation site, the Eagle office, is in Rochester, NY. While one of the evaluators resides in Rochester, herself being an Eagle employee, the other two evaluators reside in Syracuse, New York, roughly ninety miles to the east. This distance makes physical site visits by the latter two evaluators difficult, with the drive being roughly one-and-a-half hours both ways with the potential to encounter inclement weather, particularly in the winter months (i.e., when this evaluation will occur). Therefore, the evaluators are limited primarily to remote evaluation methods, except where the Rochester-based evaluator is able to operate independently or serve as an effective liaison.

· Access to research subjects: Because all stakeholders, clients, and subjects have their own day-to-day responsibilities to which they must attend (many of which are related to billable project work and deadlines), the evaluators are limited in the amount of time and effort they can request from these parties. Specifically, the main stakeholder requested no more than a half-hour at a time be taken for interview purposes and that only a select group of personnel be interviewed (five people in total, herself included). Considering this, data gathered from interviews will be drawn from only a portion of the larger group being studied in the evaluation, which may lead to conclusions that do not necessarily apply to everyone on the team. Likewise, three of the five recommended interviewees are in management positions, meaning data drawn from discussions with them may not accurately reflect the perspectives of those in the production workforce.

· Resources: Due to the academic nature of this evaluation (i.e., it is being completed as coursework), the evaluators do not have any extra resources aside from what they can provide themselves (software provided as part of tuition, self-made instrumentation, etc.). Additionally, the team consists of three first-time program evaluators, relying on their own working knowledge of the process and without outside help from additional resources.

· Information sources: As mentioned earlier, the circumstances of this evaluation require most of the data to rely on first-person testimony and word-of-mouth via interviews and survey, meaning there is little potential for results based in concrete numbers. Although the opinions and experiences of the subjects is indeed important, the lack of numeric, fact-based data may put the validity of the results into question. Another limitation is the widely varied nature of the onboarding program. Based on preliminary discussions and materials received from the clients, it is clear that onboarding to the Creative & Interactive team can be a widely varied process and often dependent on the individual; therefore, it may be difficult to identify conclusive evidence or trends.

Logistics and Timeline

Three people (evaluators) will be responsible for the implementation of this evaluation: Abdullah Albelehy, Sarah Bernhard, and Zachary Schuster. The distribution of labor will be as follows (note: these assignments are planned and therefore subject to change):

· All three evaluators will partner on the following tasks:

· Formulation of interview and survey questions

· Conducting one-on-one interviews

· Processing, analyzing, and interpreting of data gathered

· Drafting of the final report (specific section/topic assignments TBD)

· Monitoring of and adherence to budgets and timeline

· Zachary Schuster will import and disseminate the finalized survey(s) in Qualtrics and grant the rest of the team access to the results for analysis. He will also be responsible for the dissemination of the evaluation report to client stakeholders upon its completion.

· Sarah Bernhard, herself a subject-matter expert given her employment and tenure at Eagle, will review all materials prior to their implementation to ensure the approaches used will resonate with Eagle employees. Likewise, Bernhard will review the final report, in its entirety, for accuracy and consistency of terminology. She will also confirm meeting times with Eagle employees involved in the evaluation (client stakeholders, subjects, etc.).

· Abdullah Albelehy will be responsible for scheduling client and/or subject-facing interviews, partnering with Bernhard to confirm schedules and contact information for Eagle employees. He will also be responsible for coordinating and scheduling weekly team meetings.

The following timeline will be used as a guide for evaluation proceeding (note: like the distribution of labor, this timeline is subject to change):

Task	Start Date	Completion Date
Development of Survey Questions	March 4th, 2019	March 6th, 2019
Development of Interview Questions	March 7th, 2019	March 11th, 2019
Creation of Survey in Qualtrics	March 12th, 2019	March 14th, 2019
Deployment/Completion of Survey	March 15th, 2019	March 22nd, 2019
Client and Subject Interviews (virtual)	March 25th, 2019	March 29th, 2019
Analysis of Survey Data	April 1st, 2019	April 4th, 2019
Analysis of Interview Data	April 5th, 2019	April 9th, 2019
Drafting of Evaluation Report	April 10th, 2019	April 18th, 2019
Finalization of Evaluation Report	April 19th, 2019	April 28th, 2019
Delivery of Final Evaluation Report to Client	April 29th, 2019	April 30th, 2019

Evaluation Budget

The approximate cost to evaluate the Eagle Productivity Solutions onboarding process is $1,410. The SZA Evaluation Team charges a one-time $250.00 consultation fee, $22.00 per hour (detailed minutes of work will be provided) and a one-time technology fee of $500.00 for all services and deliverables.

SZA estimates that the evaluation will take 30 hours of time to complete. These billable hours include, but are not limited to, client discussions, surveys (questions, format creation, analysis etc.) interviews (semi-structured in 15 minute intervals, data collection, etc.) and other evaluation techniques. The technology fee covers any software and technical solutions used by SZA during the Eagle Productivity Solutions onboarding evaluation.