10 single-spaced pages

Collaborative Evaluations in Practice, pages 45–55. Copyright © 2015 by Information Age Publishing All rights of reproduction in any form reserved. 45

CHAPTER 4

INCORPORATING COLLABORATIVE EVALUATION

IN A DIVERSITY BUSINESS PROGRAM

Adam P. Denny, Eunkyung Na, and Michael Voris

INTRODUCTION

The concepts of collaboration and diversity go hand and hand both in the evalu- ation process and in the business environment. Being collaborative has little ad- vantage without a diversity of backgrounds, knowledge, skills, and viewpoints among stakeholders, and diversity provides few benefi ts without a collaborative environment. The purpose of this chapter is to introduce the use of a collabora- tive approach using the Model for Collaborative Evaluations (MCE) (Rodríguez- Campos & Rincones-Gómez, 2013) to an evaluation of a workplace diversity program that itself is designed to promote better collaboration within a business organization. The CEO of Benucon approached us to perform a formative evalu- ation of the fi rm’s modest program designed to enhance the nature and effective- ness of its diverse workforce.

46 • ADAM P. DENNY, EUNKYUNG NA, & MICHAEL VORIS

What do we mean by workforce diversity? Why would a fi rm like Benucon be concerned enough about it to have a workforce diversity program? Why would the company hire evaluators? Some background begins to make that clear. To busi- ness organizations, a more traditional defi nition of diversity arose in the 1960s. It had been based on the anti-discrimination legal protection established by US law and enforced by the Equal Employment Opportunity Commission (EEOC). Ini- tially this dealt with differences in terms of race, color, national origin, religion, and sex (Harvey & Allard, 2005). The legal protections were later expanded to age (those 40 years of age and above) and those with disabilities. However, the busi- ness view of diversity began to evolve after the Hudson Institute issued a report in 1987 under a United States Department of Labor contract called Workforce 2000 (Johnston & Packer, 1987). This report made dramatic projection about changes in the US workforce and the gap that would develop between the personnel needs of fi rms and the probable skills possessed by future entrants to the US workforce pool.

Data since that report have reinforced the view that organizations need to tackle the diversity issue. According to the Center for Immigration Studies (CIS), nearly 14 million immigrants entered the United States between 2000 and 2010, a record. The Center also predicts that, by 2050, there will no longer be a majority race in the United States (CIS, n. d.). Workforce 2000 and reports like the one from CIS made it clear that just complying with EEOC anti-discrimination rules was not enough. Business fi rms might avoid lawsuits with simple legal compliance, but to maintain or enhance their effectiveness and their ability to compete, they would have to proactively deal diversity in a much broader form (Ho, 2013). Given this, we decided to enter this evaluation process with the broader view of diversity that includes differences beyond social aspects such as race and nationality. It also includes other personal qualities such as qualifi cations, physical abilities, sexual orientation, attitudes, and perspectives.

Given the nation’s increasingly diverse population, fi rms had to adjust their re- cruiting strategies to reach and appeal to these highly diverse prospective employ- ees. However, just having a diverse workforce is not enough for fi rms to gain ef- fectiveness and competitiveness from it (Jehn, Northcraft, & Neale, 1999). Once they employ them, they need to retain them, which means creating a workplace environment that accepts and respects all employees. To get the most out of their workforce individually and collectively, both managers and employees need to embrace diversity and see its advantages to them and the organization. Our evalu- and Benucon, like many fi rms, recognized that they needed an active strategy to recruit, retain, and motivate a diverse workforce. At the time the CEO contacted us, their efforts in this regard were extremely limited. We were impressed by Benucon’s timely commitment to enhance a program to help create a diverse and effective workplace. We believed our evaluation would make a signifi cant differ- ence in Benucon and improve the work environment for its employees.

Incorporating Collaborative Evaluation in a Diversity Business Program • 47

THE PROGRAM

Benucon is a mid-sized company in a growing urban area. Its core business is civil engineering, transportation, and construction with services such as site plan- ning, building roads, bridges, airports, and public facilities, engineering design, inspection, and asset management. The company started as a small family-owned business about fi fty years ago. It has been growing steadily since. Benucon does a lot of business with state and local governments. The company’s mission is to deliver high quality, cost effective, and environmentally conscious services. Re- cently, the company has expanded into new United States markets outside of its home state, and is exploring the possibility of international market for its services. Over the last ten years, Benucon ranked consistently among the top 500 design and engineering fi rms by the Engineering News Records. The company employs full-time and part-time workers. Since the company added large-scale landscap- ing and maintenance to its business a few years ago, it has increased its hiring of hourly-wage workers. The company workforce at the time of the evaluation was diverse in nationality, background, qualifi cation, and abilities. However, the fi rm’s modest program lacked effectiveness to deal specifi cally with this diversity leading toward potential increases in workforce morale, employee organizational commitment, productivity, and Benucon client relationships.

WHY A COLLABORATIVE APPROACH

The MCE is an approach that guides evaluators to maximize the benefi ts of the evaluation by using six interacting concepts to achieve the desired results. Those six components are: (a) identify the situation, (b) clarify the expectations, (c) es- tablish a collective commitment, (d) ensure open communication, (e) encourage effective practices, and (f) follow specifi c guidelines. This model was chosen be- cause of its ability to guide the evaluation in a logical manner while still main- taining the fl exibility to make necessary adjustments during implementation. The MCE has many advantages as an evaluation model, and there are two reasons the MCE was especially well suited to the Benucon evaluation. First, the MCE emphasizes the need to gain collaborative input from stakeholders within the or- ganization at every step of the process. Second, the MCE has the ability to guide the evaluation in a logical manner while still maintaining the fl exibility to make necessary adjustments during implementation. The MCE has internal feedback methods that allow a lead evaluator to manage the situations as they develop. As we will illustrate here, the MCE can be used in a real world setting to assist with the evaluation.

APPLICATION OF THE MCE

In business, it takes teamwork and the close collaboration among an organiza- tion’s employees to achieve success. These employees also have potential infl u- ence or a vested interest in the outcome of an evaluation of that organization’s

48 • ADAM P. DENNY, EUNKYUNG NA, & MICHAEL VORIS

programs. With so many parties involved, a collaborative effort of the evaluation should be conducted to get a greater chance of the results being implemented. What you read in the following sections are the evaluators’ notes and insights into applying the MCE to a business environment.

Identify the Situation

Before meeting with the client, we established a team of three professional evaluators, all with previous experience in evaluating diversity programs. We set up two formal meetings for the evaluation team to meet with the fi rm’s leader- ship team at the company headquarters. The purpose of these initial meetings was to clearly identify the situation within the organization and determine what evaluation approach was most appropriate given that situation. There were many elements of the situation that needed to be identifi ed at the start of the evaluation. At a most basic level, we wanted to gain a feel for the fi rm’s physical and social environment, which would have been hard to acquire without actually visiting their offi ces. We also wanted to identify the stakeholders in the organization who had an interest in the evaluation of the diversity program. Finally, and most im- portantly, we wanted to get a clear picture of the leadership’s expectations for the evaluation and its scope. These initial meetings were also a good time to un- derstand the vision and mission of the company and see how diversity fi t in with these key elements that expressed how the fi rm defi ned its purpose.

During our fi rst meeting, we tried to fi nd out the reason the CEO wanted this evaluation conducted, what evaluation questions she had in mind, and how the company would actually use the evaluation’s results. We also tried to identify all the stakeholders, especially ones that could be effective team members if we decided on the evaluation approach. No matter what evaluation approach we ad- opted, we needed three essential members: one who had the authority to make de- cisions during the evaluation process, one with whom we would communicate if there was a need, and fi nally one who would get information and data we wanted. If we did choose to use the collaborative approach, we also wanted rank-and-fi le employees who could join as collaboration members (CMs) to voice their opin- ions and concerns.

At one point during these meetings, we sensed that the ulterior motive of the CEO’s decision to hire evaluators might have been to show off the company’s commitment to workplace diversity to potential clients. State and federal govern- ments frequently have lucrative contracts available to businesses that have the ability to show they have a diverse workforce and have active diversity enhance- ment programs in place. When we detected this type of hidden agenda of the evaluand during the meeting, we would immediately explore further to get a true picture of the evaluation objectives and value. This would also help prevent the unexpected animosity between stakeholders and the evaluators. The CEO assured us that her primary interest was improving the work environment at Benucon and making the organization more effective. The others on the leadership team con-

Incorporating Collaborative Evaluation in a Diversity Business Program • 49

curred with this view at this initial meeting and during subsequent meetings where the CEO was not present.

Between the fi rst and second scheduled meetings, we studied what we had learned about the people, organization, and their issues. Initially, we considered using the management evaluation approach as the very top person of the organi- zation requested the evaluation. However, after the second meeting, we decided to utilize the collaborative evaluation approach. A few key stakeholders we met were willing to share their perspectives and provide access to company resources. Involving them in the entire evaluation process and making the evaluation a joint effort and responsibility would create an exciting dynamic. The CMs would con- tribute substantially and help strengthen the evaluation. Yet, we the evaluators would maintain overall control of the collaborative evaluation.

We used the items in the identify the situation checklist of the MCE to guide us. The stakeholders we identifi ed were the CEO, the rest of the leadership team we met at the fi rst meeting, employees, current Benucon clients, potential national and international clients, and sub-contractors used by Benucon. We reviewed the evaluation questions established by the leadership team at the fi rst meeting. We wanted to make sure the suggested evaluation questions provoked the interest of key stakeholders, provided important information, and had the true potential to create positive impact on the client organization.

Clarify the Expectations

With the collaborative evaluation approach adopted as our methodology, we started to organize our evaluation plan. The same way that an organization needs a plan to construct a building, an evaluator needs an outline to review a product. This outline takes the form of clarifying the expectations of what is to be complet- ed. It is conducive to higher productivity and greater satisfaction. As evaluators, we often found ourselves guiding the process so that the group of collaborators stayed within the bounds of what was a productive meeting. Careful guidance and encouragement along the way led to a minimal amount of time spent on tangents. This process provided the group with a sense of pride in the creation of the prod- uct and had a higher rate of sustainment after the evaluation was completed. We considered fi ve components in clarifying the expectations: The role of the evalu- ators, the role of the CMs, the criteria and standards, the evaluation process, and the budget.

We knew we would need to clarify our roles and those of the CMs in the col- laborative evaluation process. For instance, to achieve that we planned to ask many questions to get to know the CMs and gain a greater understanding of the strengths they possessed that could be used to the advantage of the evaluation. It was important for the evaluators to have an explicit structure in place to increase the trust level from the CMs and foster a professional environment.

Often times, evaluation clients are excited about the evaluation potential and want the evaluators to explore an unrealistically wide range of questions. Con-

50 • ADAM P. DENNY, EUNKYUNG NA, & MICHAEL VORIS

sidering the budget limit, audience attention span, and evaluation manageability, a convergent clarifi cation phase is often inevitable. We expressed our judgment to reduce the number of evaluation questions. Benucon agreed with the benefi ts of limiting the evaluation questions to a few realistically manageable ones. The budgetary plans seemed to be the biggest factor that infl uenced the fi nalizing the more limited list of evaluation questions.

The CMs became the key to gaining additional information from their back- grounds knowledge and experience to add to the evaluation. Clarifi cation of their roles was an integral component of the evaluation and provided explanations and suggestions that were deemed valuable not only for their respective levels, but gave the evaluators insight into the organization as a whole. Once the evaluation questions were developed, we had to clarify in proper evaluation and method- ological techniques to get the best answers to the questions. We found that in the beginning, more direction was needed, but as the momentum built, the group and meetings began to move effi ciently. Soon, the CMs were able to collect responses themselves using the questions they had helped develop, so they had become much more vested in the process. This further led them to want to include more information and increase their section’s contribution to data collection.

Having many stakeholders with equally weighted power can make the evalua- tor’s position more tenuous at times because of the opposing decisions to be made. It is important in both large and small groups to make sure that expectations are grounded and explicit. One contentious issue was the evaluation plan. Naturally, the organization wanted to obtain results quickly, so that they start improving their diverse program as soon as possible. However, we had to continually empha- size the need to stick to the evaluation plan and all its steps before implementing changes that had not fully been evaluated. This groundwork laid the goals of the future events. Each step noted above involved coordinating meetings on multiple occasions. Between meetings, the CMs often freely shared ideas or concepts that could be brought up at the next meeting. We made sure though that each meeting fi nished with a written set of deliverables that were used to establish guidelines, procedures, or policy on set items. This step allowed for better clarifi cation during the reporting stage. From each conclusion of a section, the deliverable represented a clear-cut transition into the next section.

Establish a Collective Commitment

Benucon leadership informed us of the key CMs that would be available to work with us during the evaluation. They were the Vice President of Operations, the Human Resources (HR) manager, and an executive assistant. Needless to say, getting the CMs’ cooperation was essential in a successful evaluation. We needed to establish a good working relationship. The preferable CMs were the ones who knew the company, had access to its data, and most importantly, those personnel who could devote their time and effort to the shared evaluation vision. In other words, we wanted CMs who would be willing to take action and deliver the re-

Incorporating Collaborative Evaluation in a Diversity Business Program • 51

quested information and data on time. Also, the CMs had to be easily accessible to us. The Benucon leadership listened to our suggestions and agreed to re-constitute the CMs for the evaluation: The Vice President of Operations, the HR assistant manager, one of CEO’s assistants, and the Communication Assistant Manager. We requested Benucon to add a few employees to the team to contribute their perspectives to the evaluation process. We decided on one mainstream employee, one highly qualifi ed surveyor, and one landscape maintenance worker. The evalu- ators made sure that at least one member was bilingual in case there was a need to interpret the surveyor and the worker who had limited English speaking ability. The collaboration team of the Benucon evaluation were thus comprised of four management staff, three employees, and three evaluators.

All the CMs were part of this evaluation and decision-making process. Their contribution was essential to make a difference at Benucon. The CMs were to meet regularly. It was helpful to establish decision-making procedures. We be- lieved that maintaining a collective commitment was the key for success. How- ever, we could not ask for an all CM meeting every time there was a decision to make. We set up a protocol that we would like to follow in that respect, discussed it with the CMs, and had them agreed to it. The main contact would be the HR Assistant Manager. We also wanted to be able to bypass the main contact and go directly to the president’s assistant for certain company information that this as- sistant could deliver in timely manner. We also established a protocol to follow in case there was a confl ict to resolve. We put all these in writing and kept them in our fi le. One of the positive outcomes of establishing a collective commitment was that the CMs and evaluators got some relief from evaluation anxiety about their time and contribution.

Ensure Open Communication

The evaluation questions established for this collaborative evaluation were complex because of the diverse nature of personnel needs, the diverse engineer- ing intensive services the fi rm provided, and the lack of an established diversity program at Benucon at the time of the evaluation. Due to this complexity and the diversity of the CMs, it was important to maintain an open source of communica- tion that was multifaceted in its approach as well as effective in its means of com- munication. Each person was to bring a different perspective to the group and had different demands from the job. Ensuring that meetings were held to promote the effi cient use of time while collecting essential information from all the CMs was important to create a credible group structure that maintained active participation throughout the process.

In order to ensure effectiveness, it was necessary to establish the timelines, communication schedule, and due dates for all correspondence. With the wide array of stakeholders, it was important to establish multiple methods in which people could be involved and yet remained fl exible. Everyone in the group had the same goal in mind, and it was equally important to make sure that everyone

52 • ADAM P. DENNY, EUNKYUNG NA, & MICHAEL VORIS

understood that although the group members might see different ways of achiev- ing that goal, there was one goal that was to be achieved.

Initially, meetings were all face-to-face and extra time was planned. These initial meetings were designed to establish direction and introductions. At dif- ferent points of the meetings, personnel were called upon to bring their areas of expertise forward to discuss background information. The CMs were also asked to review different practices of theirs to see if they could shed some light on the information about diversity in their areas. A few times, the CMs did not have the information required to discuss the topic and the evaluators scheduled outside members, but those still within the organization to give some brief information and ask questions. For example, the payroll department was asked if they could research the comparisons between the lengths of employees’ stay at the company when looking at those workers that were hired on foreign visas as compared to those who were not. This allowed a broader look at how much time the company is losing for qualifi ed candidates.

After the initial face-to-face meetings with a direction established, e-mails, teleconferencing, and internet surveys were used. Being in civil engineering and construction, most CMs were frequent users of these modes of communication. It was important to establish a level of ownership on the individuals so that the CMs would feel a sense of control to maintain constant engagement. Internet surveys generally have a lower response rate, but due to picking the CMs from different areas and being a mid-size company, it was easier for them to remind co-workers to respond. Meetings were spaced out on a schedule that would accommodate some of the CMs’ heavy workloads. The CMs who attended the meeting took notes for the ones who could not attend although meeting minutes were emailed to everyone in the group.

Encourage Effective Practices

To bring out the best in the CMs, which we believed would lead to the best per- formance, we had followed the MCE procedures to promote effective practices. This was an ongoing effort throughout the evaluation. We wanted each CM to feel that every contribution would make the difference to Benucon’s future growth and benefi t them individually and collectively. We had to show our appreciation for individual differences. We had worked on numerous workplace diversity pro- grams. We were well aware that we could make a big difference in collaborative evaluation where stakeholders and evaluators from diverse backgrounds were jointly responsible. Culturally competent evaluators not only appreciate individu- al differences of CMs and stakeholders but also respect cultures represented in the evaluation. It is inevitable that culture permeates every aspect of evaluation from objectives, questions, data collection, and even data interpretation. We also under- stood that cultural competency was not explicit knowledge or skill but it could be learned. It was worthwhile to teach and encourage it by modeling. Our familiarity and ease with many different cultures and their implications on peoples’ behaviors

Incorporating Collaborative Evaluation in a Diversity Business Program • 53

had signifi cant positive effect while working with many different people from different backgrounds at Benucon. Understanding and accepting the differences also lead to creativity and harmony, which was part of Benucon’s goals that they wanted to achieve.

Follow Specific Guidelines

Quality is the process required to ensure that the project will satisfy the needs for which it was undertaken (PMI, 2013). Quality should be viewed and consid- ered heavily throughout the evaluation using guidelines that meet the needs of the organization. It could be compared closely with validity, as validity is the extent to which an evaluation tool measures what it is intended to measure. Therefore, we decided to follow the specifi c guidelines that the committee outlines with the help of the evaluators. The last thing we would want was to evaluate a program only to realize that we had measured the wrong thing.

The collaboration guiding principles (Rodríguez-Campos & Rincones-Gó- mez, 2013), the guiding principles for evaluators (American Evaluation Asso- ciation, 2004), and the program evaluation standards (JCSEE, 2011) were fol- lowed throughout this evaluation. All evaluations have a life cycle and a process adapted to them. Depending on the size of the program to evaluate, there could be a need to break down the process steps into assigned responsibilities based on those specifi c guidelines. During each stage of the evaluation, oversight is to be conducted so that the deliverables are completed on time and on point. With a well-established guideline policy in place, it will help establish these processes and the stages as they start and stop.

Remember that some processes are contingent on the previous steps taken be- fore it. When a delay happens in an early stage, it may continue to affect all of the stages after it. When dealing with diffi culties that arise, tradeoffs will have to be balanced and decided upon. This means that there can be further problems for not following the guidelines. Delays can cause the loss of time and money and could also distract the engagement of the group if they feel like the group is not progressing.

When following the specifi c guidelines, time was built into the schedule to al- low for some unforeseen problems that may arise. The evaluators usually see this in data collection as that tends to have the most unforeseen delays. Also, the CMs must decide what else is needed after they receive the results. If so, that will take some time. Are the stakeholders prepared to assign the development of the new section at the cost of delaying the implementation of the results? That is the trade- off that will have to be decided upon. Fortunately, with our evaluation the CMs were very specifi c with their guidelines and acquired the data that they needed to recommend some policy changes.

It was important to plan out each entire process before we began as the results of one process had the possibility of being linked to other processes. Each process within itself has inputs that are what the groups use to understand their directive.

54 • ADAM P. DENNY, EUNKYUNG NA, & MICHAEL VORIS

Using those inputs, they have access to tools and techniques that develop what is needed from that group and phase. The outputs are the deliverables that are tied to other processes that are needed in order to complete the work.

LESSONS LEARNED

Some of the biggest lessons we learned had to do with the time and expectations. The evaluators used their previous knowledge to build in time to make sure that deliverables were completed on the set date so that the next phase could start as scheduled. The nature of this evaluation consisted of ascertaining how to utilize the resources to its full potential. The size of the company and many people in- volved also created a need for an appropriate management of time to navigate through the personnel and their responses. It was imperative that evaluators gave suffi cient time to document, collect, and analyze responses to provide a holistic picture for the evaluation.

It was also important for the evaluators to establish and communicate realistic expectations to both the stakeholders and the person or group who hired them. During some of the meetings, there were times when people within the group would let their enthusiasm get the best of them and suggested targeting results that were unrealistic to attain. By setting the expectations, each meeting moved quickly, too. Everyone stayed on task, which lead to a continued involvement of those individuals who had a demanding job at Benucon.

We also learned that strong program knowledge and data analysis skills alone were not enough to make effective evaluators. We had to wear many hats as evalu- ators. We were the leaders and managers of the evaluation. At the same time, we also needed to act like a psychologist, nurturer, counselor, mediator, detective, and more to bring out the best from the CMs with different characters and needs. The process and experience broadened our views and enriched us. We were espe- cially appreciative of a CM, a front line worker who was in charge of landscape maintenance, for his contribution. This person brought in fresh insight from the fi eld to Benucon evaluation. We noticed that his self-confi dence level was visibly higher toward the end of the evaluation. We can produce a better evaluation if we try to understand people, their needs, aspirations, and motivation.

As mentioned in the introduction of this chapter, collaboration and diversity go hand and hand because neither is of maximum value without the other. How- ever, collaboration is not a particularly easy or intuitive process. That was why the MCE was an invaluable tool for making it happen in this formative evalua- tion at Benucon. The MCE framework provided a logical structure and detailed methodology for the evaluation. Especially in dealing with business organizations like Benucon, they expected a clear plan that was thorough and effi cient in terms of using time, resources, and personnel. Use of the MCE provided that assurance. Also, based on the continuous feedback from the Benucon stakeholders, using the MCE framework was not only valuable to the professional evaluation team, but also helped promote the buy-in of those at Benucon that would continue to

Incorporating Collaborative Evaluation in a Diversity Business Program • 55

implement the organization’s diversity programs long after the evaluation was complete.

REFERENCES

American Evaluation Association. (2004). American Evaluation Association guiding principles for evaluators. Retrieved from http://www.eval.org/p/se/in/q=guiding+principles+fo r+evaluators

CIS (n. d.). Center for Immigration Studies. Retrieved from www.cis.org Harvey, C. P., & Allard, M. J. (2005). Understanding and managing diversity: Readings,

cases, and exercises (3rd ed.). Upper Saddle River, NJ: Pearson Prentice Hall. Ho, G. C. (2013). Discriminatory diversity defi nitions: The ironic consequences of mana-

gerial diversity conceptions. (Doctoral dissertation, University of California Los Angeles). UCLA: Management (MS/PHD) 0535.

Jehn, K. A., Northcraft, G. B., & Neale, M. A. (1999). Why differences make a difference: A fi eld study of diversity, confl ict, and performance in workgroups. Administrative Science Quarterly, 44(4), 741–763.

Johnston, W. B., & Packer, A. J. (1987). Workforce 2000: Work and workers for the 21st century. Indianapolis, IN: Hudson Institute.

Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.

PMI Standards Committee. (2013). A guide to the project management body of knowledge: PMBOK Guide (5th Ed.). Project Management Institute: Alibris.

Rodríguez-Campos, L., & Rincones-Gómez, R. (2013). Collaborative evaluations: Step- by-step (2nd ed.). Stanford, CA: Stanford University Press.

PART II NONPROFIT SECTOR

Co py ri gh t 20 15 . In fo rm at io n Ag e Pu bl is hi ng .

Al l ri gh ts r es er ve d. M ay n ot b e re pr od uc ed i n an y fo rm w it ho ut p er mi ss io n fr om t he p ub li sh er , ex ce pt f ai r us es p er mi tt ed u nd er U .S . or a pp li ca bl e co py ri gh t la w.

EBSCO Publishing : eBook Collection (EBSCOhost) - printed on 10/15/2019 12:36 PM via UNIV OF SOUTH FLORIDA - MAIN S3654090 AN: 959354 ; Rodriguez-Campos, Liliana.; Collaborative Evaluation in Practice : Insights From Business, Nonprofit, and Education Account: s3654090.main.eds