ogl 540 final project.

Design

We have decided to utilize a quantitative research design. We are going to be using a longitudinal experimental design. Our participant group was chosen through a multistage procedure in which we first sought permission from Ironwood Counseling to conduct the experiment and then selected participants based on their employment within the organization. We chose to focus on the administrative employees at Ironwood Counseling to create our sample group.

We are utilizing a pre-experimental, one-group (within-group) pretest-posttest design to study the effects of external motivators on long-term employee engagement levels. One group of participants is being studied and will serve as both the control and test group. We will conduct a measurement, implement the external rewards, measure after external rewards, and follow up 6 months after that for a total of three measurements per participant.

Group A O----XO-------O

Participants

We will be using a multi-tiered convenience-based sampling method for choosing our participants. We started by selecting the organization to participate in the experiment, Ironwood Counseling. This organization was selected for a number of reasons. The researchers had familiarity with leadership at Ironwood Counseling, making it easier to obtain permission to conduct the research. As well, Ironwood Counseling did not already have an external rewards program in place, making it possible to use it as a site for implementing the rewards in an experiment. From there, we chose to use the administrative staff for the experiment based on the number of employees as well as the ability to set criteria for performance that the external rewards can be tied to.

All participants will be assigned to the experimental group. The experiment consists of surveying initial employee engagement levels and then applying the external rewards and measuring subsequent engagement levels in the same population.

Using the Power Analysis Sample Size calculator by AI-Therapy Statistics, we determined that 27 participants are needed to obtain meaningful data. This is based on the sample group using the same subjects, with a significance level of .05 and a power level of. 8 with a significant standard deviation being .5 for effect size (AI-Therapy.com, Sample size calculator).

Data collection procedures

Data collection will be accomplished through surveys. An initial survey will be distributed assessing the employee's thoughts and feelings surrounding external motivators and their current perceived productivity and engagement. Participants will respond to questions on a Likert scale to indicate their agreement with statements regarding their company, motivation levels, productivity, and engagement. Six months after the employees received an external motivator from their employers, employees will be asked to complete the same survey assessing employees' thoughts and feelings surrounding external motivators and their perceived productivity and engagement. Participants will be given a unique identifier to ensure participation but to maintain anonymity.

For this experiment, we plan to use the Employee Engagement Scale (EES), a device designed by Shuck, Adelson, and Reio to measure employee engagement along three dimensions: cognitive engagement, emotional engagement, and behavioral engagement (Shuck et al, 2017). Developed in consultation with experts in employee engagement, the EES has been tested for construct validity against other reputable workplace engagement scales. The developers refined the tool for duplication of questions and found similar results in the use of the EES when compared with other validated scales.

We will also be asking basic demographic questions to account for tenure within the organization as a potential intervening variable. Other mediating variables (job stability and external job opportunities) will be measured through a scaled response of participants' perceived beliefs.

Procedure Steps

1. Participants will be sent an initial email to inform them of their company's participation in a research study. While their participation is not mandatory, the initial email will thank them for their consideration of participating. In that email, they will be notified that the responses recorded will be kept confidential and only reported in aggregate form to protect anonymity. Participants will be informed that the research is being done to better understand employee engagement.

2. Participants will be sent a link to the pre-treatment survey. Each survey will be tied to a unique identifier so that future scores can be matched to show changes over time. A list of all participants and their identifiers will be maintained by the researchers and used as the participant's passcode for accessing the survey. The survey itself consists of twelve questions to determine employee engagement as well as additional questions to gauge job tenure, perceived job stability, and perceived external opportunities.

3. Following the completion of the survey, the workplace will implement external rewards, such as a monetary bonus or extra time off for a six-month period. Criteria will be given to the organization for when different rewards will be given to ensure that rewards are given in a fair and consistent manner to all participants.

4. After six months, participants will be sent an email with their unique identifier and a link to a duplicate of the pre-treatment survey to assess their new engagement level. Following this survey, the experimental external rewards will cease.

5. Six months after the experimental external rewards have stopped, a final survey will be given. This will allow the researchers to assess if use of external rewards impacts baseline engagement levels by comparing initial scores with post-scores. Participants will be thanked for their participation in the surveys.

6. The results from the three surveys will then be compared to determine if implementing external rewards leads to increased employee engagement and whether that increase is sustained after the external rewards are no longer available.

Data Analysis Procedures

Data will be reviewed for outliers and statistical anomalies. Each participant will be assigned a unique identifier so that the level of engagement can be tracked on an individual level. The unique identifier will only be known to the experimenters to ensure honest answers from the participants. The goal would be to determine how many employees' engagement went up or down during the time period. Then the data would be combined to give an overall view of engagement change across the organization. Both supportive and contrary findings will be explored.

We will analyze the results for the average change in each of the different engagement questions as well as the change in engagement for each area of engagement (cognitive, emotional, and behavioral). Results from the Pilot Study indicated a discrepancy in scores between the cognitive, behavioral, and emotional subsections of engagement that will need to be analyzed for potential impact on overall long-term engagement. As well, we will look at the standard deviation in the responses to determine if the change seen between responses is statistically relevant.

A comparison of responses both between the employees' responses at various points in time as well as across the group as a whole will be conducted. The responses at various points in time can indicate the overall effect of external rewards on employee engagement. The comparison across the group as a whole will allow us to check our mediating variables of job tenure, perceived job stability, and perceived external opportunities against the changes in engagement levels that we see. This is important as previous research has shown that those mediating variables can influence the impact of external motivators and it is important that this experiment accounts for that influence.

Validation Approaches

There is potential for mortality to be an internal threat as employees may leave the company during the time period of the experiment. We will account for that by including a discussion of similarities that those who drop out of the study have in contrast with those who were still involved at the end of the study. If there are common factors in responses to questions surveyed, there may be potential for additional research into that area as a factor in employee turnover.

Another possible threat to internal validity in this study is testing. Because the survey will be administered three times, there is a risk that participants will become familiar with and remember survey questions and put the same answers each time they participate in the survey. It is our hope that by conducting the testing at six-month intervals, there is less risk of participants memorizing previous answers. In addition, the survey questions will be randomized to ensure participants are not answering questions in a predetermined pattern.

It is possible that an external validity threat related to the participant selection exists. We may not be able to extrapolate the data to a different type of employee. Since we are focusing on administrative office staff, the same results may not hold true for employees in more physically demanding jobs or who require a lot of travel for their work. We will also use the demographic data to determine if our results can be extrapolated to more diverse groups.

Methods to validate the data will include excluding surveys that mark the same response for every answer and including questions to check for responder honesty (asking the same question with responses flipped). We will also exclude outliers that may identify an individual employee both for confidentiality and data validation purposes.

Ethical Considerations

Privacy of employees responding to the survey is important. We want to ensure that participants are comfortable being honest about their employers or we will end up with skewed results. Employees will be designated a unique identifier that will ensure anonymity from employers. They will indicate their unique identifier as a response to one of the survey questions. This will ensure that employees are participating in the survey without revealing the identity of the participant.

We believe it is important that all employees participate in external rewards for the sake of fair treatment in the workplace. Imposing external rewards on one group and not another could lead to decreased engagement and performance in the control group which is why our design has eliminated a control group and applied the experimental variable equally to all participants.

A potential point of concern for ethical considerations is the voluntary participation of our research participants. As we have selected the organization to participate, Ironwood Counseling, our goal is to have all administrative staff participate in the experiment based on convenience sampling. However, while the organization has consented to participate in the experiment, the participants may feel pressure to participate as part of their employment. We will be making it clear within the consent form that if they do not wish to participate in the experiment, they have the option to decline to participate and they will not need to take the surveys. Their employer will not be informed of those participants who decline to participate to avoid any potential pressures from leadership to have the employees complete the survey. This will ensure all participants give explicit consent for their participation.

The experimental procedure of applying external rewards will be done regardless of consent to participate in the survey as the rewards will be given by the supervisor at their discretion. As well, we do not want participants to feel pressured to participate in the research in order to receive monetary bonuses or vacation time. By not tying the rewards to participating in the three surveys, participation will be voluntary and uninfluenced by the rewards.

Pilot Data

Data Collection

The pilot survey was sent out to ten participants on November 23, 2020. Responses were collected through November 25, 2020. The survey was collected online through Google Forms. Once a survey was completed by a participant, their responses were immediately available for review.

Participants for the pilot survey were recruited based on convenience sampling through Ironwood Counseling in Gilbert, Arizona. Participants were informed that the purpose of the survey was to better understand people's perceptions of their engagement at work. Once they had consented to participate in the research, they were sent a link to the survey along with directions and expected timeline. Participants had roughly 48 hours to complete the survey in order to have their responses recorded.

Data Content

Data was collected for several variables. A single question was asked for each of the mediating variables: tenure perceived job stability, and perceived external job opportunities. The dependent variable in our experiment is employee engagement. For our pilot data, we used the Employee Engagement Scale, an engagement questionnaire consisting of 12 questions centered on the cognitive, emotional, and behavioral components of engagement to provide a comprehensive engagement score (Shuck et. al, 2017). Each of the engagement components is assessed with four questions to compose an overall subset score as well as an overall engagement score. The purpose of collecting the pilot data was to see if there were any unexpected results in the survey that required further review of literature before instituting the full experiment. As well, our pilot study was conducted to check for accessibility and usability of the survey by participants.

Data Management

The data that was collected, organized, and managed in Google Forms was only accessible to those controlling the experiment. Employees and managers at Ironwood Counseling were not able to view the results of the survey. After responses were recorded, Google Forms created a bar graph with responses for each question. The years that each participant had been with the company was also recorded and a pie chart was made with the results. Refer to Appendix A for examples of the chart and graphs that were created after responses were recorded.

Data Analysis

The mean and standard deviation for each of the questions on the engagement scale was calculated, as well as on questions for two of the mediating variables, perceived job stability and perceived external opportunities (see Appendix A).

An interesting result that came out of this pilot study is that emotional engagement scored a lower average (3.7) than cognitive (4.3) or behavioral engagement (4.35). On the Employee Engagement Scale, emotional is "defined as an employee’s intensity and willingness to invest emotionality toward positive organizational outcomes" (Shuck et. al, 2017). Knowing that emotional engagement earned a lower score in the pilot will allow us to track and monitor similar discrepancies in the engagement scores between the three dimensions throughout the experiment.

Implications

The aim of our research is to test the influence of external motivators on long-term employee engagement, with the mediating variables of tenure, perceived job stability, and perceived external opportunities. The pilot data revealed the average responses on the mediating variables perceived job stability (3.5) and perceived external opportunities (3.7), a lower average than initially anticipated when designing the experiment.

With the knowledge of the current financial climate impacting many companies globally, these scores may be unnaturally influenced in ways that would have been atypical a year ago. The global COVID-19 pandemic has caused many employees to feel that their jobs are less secure and that the job market is unstable. We will need to more carefully monitor the impact of the mediating variables on long-term engagement to make sure the final results are not reflecting their influence rather than the effects of external motivators. As the end of the current pandemic and subsequent financial impact is unknown, controlling for these variables in participants is impossible. Analysis of the results will need to include a discussion of how market fluctuations may have impacted these scores.

Discussion

Based on the data that was collected in the Pilot Study, the mediating variables we have identified (tenure, perceived job stability, and perceived external opportunities) influence the motivation of employees that the survey measured. It will be important for us to monitor those variables to ensure that they are not influencing the primary data that we will collect. They could be better monitored by adding additional questions to assess the influence of the mediating variables. Adding additional questions will help us to gauge the impact that these mediating variables have on employee engagement and control for their influence.

Research on employee engagement and motivation is becoming increasingly more important as organizations are changing in response to the COVID-19 Pandemic. The traditional workplace may no longer be applicable to many industries. Based on this and other research on the effects of external rewards on employee engagement and motivation, employers will be able to understand the importance and possible necessity of implementing external rewards in their workplaces to ensure that their employees are engaged in the work that they are doing despite the changes occurring in the workplace.

Though the results of the pilot study and review of relevant literature indicate a positive relationship between external rewards and long term employee engagement, further research is needed to determine if these results can be generalized to larger corporations and companies where resources for providing external rewards are not available.

Limitations

The research design does place some limitations on the scope of the research. We chose to conduct the study on office workers at a mental health facility out of convenience and accessibility to the audience. It is possible that the results of the pilot data and the subsequent long-term study to follow would not be generalizable outside of that type of employee class. Further research would need to be done to determine if the results are generalizable to a wider audience such as larger corporations who may have a more substantial number of employees and more resources to provide external rewards.

Another limitation is the difficulty in replicating the results of the long-term study. Researchers may find it difficult to implement a year-long study within an organization or set of organizations as the amount of time can be limiting. Additionally, the cost of this research can be expensive as providing the external rewards of monetary bonuses or extra vacation time holds a financial cost that may limit the number of organizations able to participate in such research. Finding an organization that would be willing to front the cost of the external rewards in order to participate in the research to better support their employees would be an important step for future researchers.

We found it important that all participants in the research be part of the experimental group to prevent any potential lowering of engagement levels in a potential control group that would see other employees receiving rewards while they did not. The psychological impact of having a control group could significantly skew results within that group and therefore alter the results of the experiment. However, this design does make the research potentially biased and thus, limited in its scope as there may have been some other factor impacting the engagement levels of the entire group that were not accounted for.

Another limitation of the research is that all participants will receive the same external rewards. Future research that could play an important role in better understanding how external rewards influence engagement could involve giving different tiers of rewards to employees. Creating a tiered reward system is something that requires further research. In giving different tiers of rewards, researchers would be able to determine if there is a minimum amount that needs to be offered as an external reward to positively impact engagement. The possibility exists that too small an external reward could result in a negative influence on engagement levels. Another possible implication of a tiered reward research structure is that there may be a cap to the level of external reward that will affect engagement levels. If there is a 'ceiling' on how much external rewards can influence engagement levels, it would benefit companies to know that as they want to be sure they are receiving a return on their investment of the external rewards. Applying external rewards beyond the ability for them to impact engagement and subsequent performance may not be cost-effective for many organizations.

The last limitation is the need to determine the history of external rewards in the industry or organization. If external awards have been regularly given by an organization, their effectiveness may wane as they become part of the normal work environment. If external rewards are a norm, their efficacy will not be equal to those at an organization who has no history of external rewards. Additionally, in an industry such as sales, commission and bonuses are commonplace therefore possibly reducing the value of additional external rewards to the employee.

Conclusion

Employee engagement is not solely determined by one factor. It is heavily influenced by many factors with external motivators only being one. While previous research has shown that external motivators can impact employee engagement, there is still an unknown relationship on the long-term effect. Accounting for all of the possible factors that may influence engagement in research requires careful measurement of job tenure, perceived job stability, and perceived external opportunities as they all play a role in initial and long-term engagement levels. In the current economic climate, it is imperative that successful strategies for long-term engagement are studied to benefit organizations globally. High employee engagement levels can lead to increased job performance and less turnover, both factors that save companies money and lead to higher job satisfaction.

It is important for companies to individually determine if providing external rewards to increase employee engagement is something that the company has resources for and if it will be effective for each company. The cost of providing extra vacation days, monetary bonuses, and other external-reward perks may not be something that a company has a budget for. These companies need to determine if changes need to be made to allow for these external rewards or determine other strategies that can be used to motivate their employees.

Employers have an opportunity to influence an increase in the engagement and job performance of their employees. External rewards are one of the ways that they can do this. Based on this and other research on the topic, external rewards are an effective way for employers to create an environment at their companies that foster long term employee engagement and job satisfaction.

References

Aliekperova, N. (2018). Motivating Factors Effecting Work Efficiency of Employees in Ukrainian Pharmaceutical Sector. Economics and Sociology, 11(1), 61-74. doi:10.14254/2071-789X.2018/11-1/4

Bin Hu, Zhenhu Hou, Miranda Chi Kuan Mak, Sabrina Lingxiao Xu, Xuhua Yang, Tianlong Hu, Yong Qiu, & Yueran Wen. (2019). Work engagement, tenure, and external opportunities moderate perceived high-performance work systems and affective commitment. Social Behavior & Personality: An International Journal, 47(5), 1–16. https://doi-org.ezproxy1.lib.asu.edu/10.2224/sbp.7353

Caldwell, D. F., O'Reilly, C. A., & Morris, J. H. (1983). Responses to an organizational reward: A field test of the sufficiency of justification hypothesis. Journal of Personality and Social Psychology, 44(3), 506-514. doi:http://dx.doi.org.ezproxy1.lib.asu.edu/10.1037/0022-3514.44.3.506

Cerasoli, C. P., Nicklin, J. M., & Ford, M. T. (2014). Intrinsic motivation and extrinsic incentives jointly predict performance: A 40-year meta-analysis. Psychological Bulletin, 140(4), 980-1008. doi:http://dx.doi.org.ezproxy1.lib.asu.edu/10.1037/a0035661

Chiu, H. (2018). Employees’ Intrinsic and Extrinsic Motivations in Innovation Implementation: The Moderation Role of Managers’ Persuasive and Assertive Strategies. Journal of Change Management. Vol. 18, Issue 3.

Comeau, G., Huta, V., Yuanyuan, l., & Swirp, M. (2019). The Motivation for Learning Music (MLM) questionnaire: Assessing children's and adolescents' autonomous motivation for learning a musical instrument. Motivation & Emotion. Vol. 43(5). pp. 705-718.

Delaney, M., & Royal, M. (2017). Breaking Engagement Apart: The Role of Intrinsic and Extrinsic Motivation in Engagement Strategies. Industrial and Organizational Psychology, 10(1), 127-140. doi:10.1017/iop.2017.2

Eisenberger, R., Pierce, W. D., & Cameron, J. (1999). Effects of reward on intrinsic motivation—Negative, neutral, and positive: Comment on deci, koestner, and ryan (1999). Psychological Bulletin, 125(6), 677-691. doi:http://dx.doi.org.ezproxy1.lib.asu.edu/10.1037/0033-2909.125.6.677

Linz, S. (2003). Motivation and Reward: A Case Study of Russian Workers. Problems of Post-Communism. Vol. 50(2), pp. 44-55. ISSN 1075–8216.

Putra, E. D., Cho, S., & Liu, J. (2016). Extrinsic and intrinsic motivation on work engagement in the hospitality industry: Test of motivation crowding theory. Tourism and Hospitality Research, 17(2), 228-241. doi:10.1177/1467358415613393

Sample size calculator. (n.d.). Retrieved November 21, 2020, from https://www.ai-therapy.com/psychology-statistics/sample-size-calculator

Shuck, B., Adelson, J., Rieo, T.J. (2016). The Employee Engagement Scale: Initial Evidence for Construct Validity and Implications for Theory and Practice. Human Resource Management. Vol 56(6) pp. 953-977. DOI:10.1002/hrm.21811

Skinner B. F. (1982). Contrived reinforcement. The Behavior analyst, 5(1), 3–8. https://doi.org/10.1007/BF03393135

Appendix A

Appendix B

The scale's internal consistency is .83 as measured using Cronbach’s Alpha Scale.