10 single-spaced pages

Collaborative Evaluations in Practice, pages 35–44. Copyright © 2015 by Information Age Publishing All rights of reproduction in any form reserved. 35

CHAPTER 3

UTILIZING COLLABORATIVE EVALUATION IN A FINANCE

AND BANKING CORPORATION Diep Thi Nguyen and Thanh Vinh Pham

INTRODUCTION

This chapter focuses on how we used the collaborative evaluation approach to evaluate a retail banking services program in a fi nance and banking corporation. The Model for Collaborative Evaluations (MCE) (Rodríguez-Campos & Rinco- nes-Gómez, 2013) was applied as the framework that guided our evaluation pro- cess. We utilized this model to actively involve our collaboration members (CMs) to conduct an evaluation for the improvement of their program.

We will begin our chapter with a brief introduction on the program we evalu- ated and the advantages of applying a collaborative approach to our evaluation. As discussed earlier, when the highest level of organizational evaluation maturity is reached, the pursuit of an organization’s value and quality goes beyond perfor- mance measurement and monitoring. Because the MCE is a comprehensive and extensive model, we will only emphasize on elements that we found the most effective and appropriate to our evaluation context. Specifi cally, we will discuss how we utilized components in the MCE to attain evaluation organization level

36 • DIEP THI NGUYEN & THANH VINH PHAM

in this program. In order to ensure confi dentiality of the entity evaluated, the fi cti- tious program name, Services, will be used throughout the chapter.

THE PROGRAM

Services was recently developed and operated by the personal banking services department in an international fi nance and banking company. The program evalu- ated was a part of a regional branch in the Asian-Pacifi c area. Based on its recent market research in the local market, the branch planned to expand its services by launching Services. This program aimed to offer banking services for both small enterprises and individual customers. The branch and the department only had experience doing internal evaluation, but were willing to employ an external, col- laborative evaluation to improve Services due to the limited success of previous evaluations.

WHY A COLLABORATIVE APPROACH

In order to conduct a sound evaluation, it is essential to have a broad sense of the characteristics of the entity we evaluate. It is not easy, however, to gain a deep understanding of our evaluand setting and context at the beginning of the evalu- ation. Poor or incomplete information may impact our decisions and ability to conduct a successful evaluation. The collaborative approach allows evaluators to make adjustments during the evaluation process and resolve unexpected issues in a timely manner. Among the various approaches we have employed, we prefer the collaborative evaluation approach, especially when it becomes clear that our cli- ent is willing and committed to work openly and cooperatively with stakeholders at a substantial level.

Collaborative evaluation requires regular interactions and communications that engender a thorough understanding between evaluators and stakeholders. In our evaluation of Services, our collaboration with the CMs (including the client) was developed and enhanced based on a mutual benefi t of the overall goal: to evaluate how the retail banking services functioned in order to strengthen its qual- ity as consequently promote the company’s competitiveness and reputation. Upon explanation of the evaluation purposes and intended uses, together, we achieved a desired level of collaboration. By analyzing the potential consequences of the evaluation as well as the stakeholders’ involvement, we were successful in mini- mizing their anxiety about the evaluation. It is important to note that after a high level of collaboration and engagement with our key stakeholders was achieved throughout the process, our evaluation went beyond merely determining the qual- ity of Services. Specifi cally, when the CMs were fully aware of the need, impor- tance, and effectiveness of the program evaluation, they considered it as an indis- pensable part of both their program and the corporation’s plans and activities. As a result, their overall organizational performance improved on an ongoing basis.

Utilizing Collaborative Evaluation in a Finance and Banking Corporation • 37

APPLICATION OF THE MCE

The business sector, particularly the banking and fi nance area, is a dynamic and highly competitive environment. Prompt and valuable feedback from the evalu- ation therefore is necessary to help stakeholders enhance the program’s strengths and work on its weaknesses in a timely manner. As a result, the program’s com- petitiveness and reputation are also enhanced.

In our evaluation of Services, the quality of banking services and customer convenience are complex to measure. We believe that we could not develop ap- propriate methods to evaluate those criteria if we had not gained the collaboration of the CMs. Their knowledge and experience in retail banking services and active engagement in the process greatly contributed to the success of our evaluation. Detailed and useful instructions for each step of the MCE helped us achieve effec- tive and active engagement with the CMs throughout the evaluation and enabled the program to obtain the high level of organizational evaluation.

Identify the Situation

As emphasized in the MCE, this component plays a fundamental role in the success of any evaluation. Although all of the subcomponents in this component were important and helpful, in this chapter we will only focus on the ones we found most useful in this specifi c evaluation: identifying stakeholders and the CMs, evaluation scope and questions, and critical evaluation activities.

Making a favorable fi rst impression was one of the critical factors highlighted in the fi rst subcomponent. It is apparent that we can only collaborate with our client and key stakeholders when an open and supportive atmosphere is created. One of our goals of the preliminary meeting was to determine if this approach was appropriate for evaluating Services. To stimulate the conversation, we asked about the background of the key stakeholders and prepared for potentially hard questions or unforeseen situations. Knowing our client’s quantitative background and highly ranked position, we focused our presentation on key points with visual aids and descriptive information. We collected information from all possible re- sources to have a holistic view of the branch and the program. We believe that our careful preparation for the fi rst meeting, knowledge and experience in banking and fi nance services, and our appropriate etiquette helped us achieve a positive fi rst impression and gain an opportunity for a fruitful relationship with our client thereafter.

During the fi rst visit, our most important objectives were to achieve a broad sense of the program’s characteristics, understand the corporation and department working culture, and identify our client’s needs, expectations, and effect on this program. We had the opportunity to get to know our client and key stakeholders, the department, and the program resources. We asked for previous evaluations or appraisal reports, and were provided information related to its evaluation ques- tions and results. We also requested our client’s initial draft of evaluation ques-

38 • DIEP THI NGUYEN & THANH VINH PHAM

tions and reviewed them with our client and key stakeholders. We considered the consistency of that draft with our client’s needs, and the feasibility and manage- ability of those questions with the available resources of the department. This step is fundamental because based on our experience, clients often have more ques- tions than the evaluation can answer. They usually think of needs fi rst rather than resources. In addition, as we attempted to identify the evaluation scope, we real- ized that it was essential to ask our client about the intentions to use the evaluation results and our client’s actions if the results were different from his/her expecta- tions. These questions helped us recognize the underlying purpose of the evalu- ation and address any potentially biased or unanswerable questions. Information gathered from discussions with our client and key stakeholders, together with our observation and analysis of the context, allowed us to determine the feasibility of applying the collaborative approach in this evaluation. This information, along with our client’s and key stakeholders’ commitment to fully engage in the evalu- ation process at the end of our fi rst meeting enabled us to move forward with this approach.

Another aspect of this component that we believe made our collaboration with our client and key stakeholders successful was the selection of the CMs. Effec- tive and active engagement of the CMs during planning, executing, and reporting phases greatly contributed to the success of our collaborative evaluation. We pro- posed a list of desired characteristics of the CMs to our client and key stakehold- ers, including but not limited to: readiness for collaboration, thorough knowledge and involvement of the program, basic evaluation skills, and good relationships with our client and other employees in the program. Working with our client and key stakeholders, we determined the CMs. Our experience was that the selection of the CMs can be fl exible as long as related changes in the selection are benefi - cial for the collaboration throughout the evaluation. In our case, a senior personal banking consultant and also one of the founders of Services were added after initial selection to enhance the capacity of our collaboration.

Following the MCE guidelines, we classifi ed critical activities into categories in a work breakdown structure chart. In this chart, we assigned specifi c respon- sibilities for evaluators and the CMs with their appropriate levels of control for these activities. Specifi cally, we jointly worked with the CMs to determine the duration of our evaluation and identify major evaluation activities with specifi c timeframes. By taking this step, everyone was clear about which evaluation ac- tivities needed to be completed and who was responsible for leading the tasks. One step we found especially helpful was gathering feedback from our CMs on a regular basis. Using a previously agreed-upon system including meetings, emails, and online surveys, we could focus and keep updated with the critical activities, and resolve unexpected problems.

Utilizing Collaborative Evaluation in a Finance and Banking Corporation • 39

Clarify the Expectations

Clarifying the expectations is a key factor that helps people involved in a col- laborative evaluation understand which issues should be addressed and search for the most effective ways to achieve the desired results. When the expectations of people involved in an evaluation process are clear, evaluators and the CMs can anticipate benefi ts and constraints of every evaluation decision. As a result, this can alleviate confl icts and enhance productivity, satisfaction, and trust among them.

In the fi rst meeting, we addressed the expectations of the CMs regarding their roles, needs, and responsibilities as well as ours during the evaluation process. We clarifi ed our role and emphasized that we were always open to communication and recommendations for improvement. We started with the role of each CM by introducing the activities involved in the collaborative evaluation. We then col- lected information on the CM’s qualifi cations and expectations, as well as their evaluation needs and suggestions for improvement. We paid close attention to what the CMs conveyed in order to fully understand their expectations. We also discussed with the CMs, one by one, how their knowledge, skills, and experience could be valuable contributions to the collaborative evaluation as well as how their roles might match expectations to reach the evaluation goals. We believe that the CMs best understand their own strengths and weaknesses. Therefore, assign- ing the CM roles based on individual willingness, availability, and accountability stimulate their motivation and maximize individual potential, thereby creating a sense of shared ownership of the collaborative evaluation.

After the client established the evaluation questions, we worked together with the CMs to clarify the evaluand criteria and standards. In order to keep the CMs focused on evaluation questions and associated criteria and standards, we pro- vided questions with implicit criteria highlighted. Everyone reviewed the infor- mation to ensure that all of us understood it in the same way. Specifi c criteria and standards were discussed and modifi ed to make the evaluation questions feasible to answer. The minutes of our collaboration meeting with focus on evaluation questions and purposes was sent to the CMs and specifi c stakeholders to ensure that everyone shared a similar understanding of the evaluation. This document also emphasized the important role of the CMs to actively and effectively engage in the evaluation process to make it reach its predetermined target.

Once the evaluation questions, criteria, and standards were clearly and for- mally agreed upon by everyone involved, we developed an evaluation plan with specifi c milestones. We also clarifi ed the three phases of our evaluation process including planning, executing, and reporting with detailed activities for each phase. We discussed this plan with the CMs to ensure we agreed on what should be accomplished for the evaluation to achieve predefi ned goals. One of the most challenging topics of this component was clarifying the evaluation budget. For instance, we emphasized with the CMs that the clearer we were about our roles

40 • DIEP THI NGUYEN & THANH VINH PHAM

and responsibilities for the evaluation procedures, the less time, energy, and re- sources we would need to use, and as a result, the less budget for evaluation would be needed to spend. We provided our client with support scheduling tools, such as a milestone chart and network diagrams, to visually display evaluation activities and timelines. Every change or modifi cation of our evaluation plan was required to be reported to the appropriate parties to make adjustments accordingly.

Establish a Collective Commitment

A collective commitment is essential to promote the intrinsic motivation of people involved in the collaborative evaluation and take responsibility for it. This commitment also fosters active and genuine engagement of people involved in the evaluation process, which is an advantage and unique characteristic of the col- laborative evaluation approach. In our evaluation, the collective commitment with the CMs (including the client) was established and achieved by our shared vision of the evaluation and mutually benefi cial evaluation purposes.

Once our evaluation plan was approved, we identifi ed and clarifi ed with the CMs the evaluation vision to share a common understanding of it. As proposed at the beginning of our evaluation, the purpose of using its results was to improve the effectiveness and effi cacy of the program, and as a result, enhance the com- petitiveness of their retail banking services. Therefore, the CMs who were directly involved and responsible for the development and operation of this program were able to benefi t from its improvements. That reward motivated the CMs to work for the best quality of our evaluation.

After determining with the CMs the shared vision of the evaluation, we intro- duced the recommendations for the positive actions and asked for the approval of our client and CMs to drive our evaluation activities in an open and supportive atmosphere, which is vital for effective practices. The CMs were enthusiastically engaged in every step of the evaluation. They provided helpful information, such as where the data would best be acquired, which time was suitable, and which way was effective to contact and interview customers because they had a good understanding of their customers’ characteristics.

Since the CMs were actively engaged in this collaborative evaluation from planning to executing and reporting, they were fully aware of its credibility and importance when it reaches the evaluative organizational level; that is, evaluation is not only a performance measurement, but also a means for other high-level organizational activities such as decision-making, strategy-making, and planning. With this in mind, the likelihood of our clients actually using the evaluation re- sults was high.

Ensure Open Communication

Open communication is a critical element for any highly collaborative activity. In this evaluation, we encouraged the CMs to share their ideas and feedback be-

Utilizing Collaborative Evaluation in a Finance and Banking Corporation • 41

cause an active and on-going communication between evaluators and stakehold- ers leads to stronger evaluation design, enhances data collection and analysis, and results that stakeholders understand and use (Rodríguez-Campos, 2012a, 2012b). During the evaluation process, we always showed our willingness to listen to comments and recommendations for improvement. For example, after our fi rst meeting with the CMs, one of the evaluation purposes was proposed as evalu- ating the customer convenience and satisfaction. According to Ferreira, Santos, Rodrigues, and Spahr (2011), the quality of banking services and customer con- venience are intangible, and depending on the context, managers may have differ- ent views on certain criteria. This idea of intangibility made it critical to convey a clear message of the evaluation process in order to make valuable judgments on Services. During our meeting with the CMs, we witnessed their various view- points on how to judge the customer convenience and satisfaction of personal banking services. Subsequently, we collected their ideas about proper measure- ment criteria via a quick survey. Then, we discussed with them in a supportive and constructive manner how to interpret the evaluation results. Finally, we came up with common weighted criteria upon which all of us agreed. Thus, everyone was comfortable and encouraged to actively engage in the evaluation process, because their ideas were taken into consideration.

Encourage Effective Practices

Encouraging effective practices in the collaborative evaluation is a key factor contributing to its success. Therefore, it is very important to establish procedures to encourage these practices so that our evaluation can reach predetermined goals. As mentioned in the clarify the expectations component, we emphasized with the CMs at the beginning of our evaluation process that effective practices were the best way to enhance productivity and minimize expenses for our evaluation.

In our specifi c evaluation of Services, we found that the teaching by example and encourage fl exibility and creativity subcomponents were the most helpful for facilitating our ongoing collaborative evaluation activities. Two of the character- istics of our evaluation in this fi nance and banking corporation are particularly notable in this context: the high level of responsibilities and seniority of the CMs, and their professional working environment. We tried to create opportunities to interact and share their ‘examples.’ Through listening to their ideas and solutions in those examples, we were able to identify everyone’s strengths, as well as learn new skills from each other.

As leaders and managers of the Services evaluation, we always tried to perform and collaborate with the CMs to the best of our ability. One thing we always bear in mind is that we only inspire others to work to their best capacity and capability when they see us work similarly as well. We not only fulfi lled the responsibilities assigned in our evaluation plan with our highest efforts, but also appreciated oth- ers creative ideas and helped pursue them.

42 • DIEP THI NGUYEN & THANH VINH PHAM

Although the evaluation plan and specifi c guidelines were proposed in the early stages of our evaluation, we aimed to provide a fl exible environment for the CMs, to encourage their creativity, and to enhance the effectiveness of our collabora- tion. Moreover, understanding individual differences of others and appreciating those differences helped us thoroughly understand each member and determine the most effective ways to collaborate with them during the evaluation process.

We also found the use of appreciation remarks was very helpful in connect- ing the CMs and us. We realized different ways to appreciate the CMs and other stakeholders so they would feel welcomed and valued. For instance, we always expressed our appreciation verbally as well as via email, and postcards. It should be noted that this practice of appreciation was not popular in the evaluand’s work- ing environment at the time we evaluated Services. One possible explanation may be based on their culture where people do not often explicitly praise other people. Although everyone in Services and the branch might know that reputation of cor- poration plays a critical role in attracting money and using of its services from individuals, they did not pay careful attention to use appreciation to enhance their reputation. We still remember one time we knew that our client received an email from a customer with outstanding remarks about the service he received from a Services staff. We discussed with our client about this intangible credibility/qual- ity of customer services in the retail banking and fi nance area and emphasized the importance to recognize and encourage this type of performance in the whole branch. Our client then decided to hold a tea party with all employees in Services program as corporation’s appreciation to his excellent performance. At our fare- well party, our CMs shared with us that our ongoing acknowledgement of their work and efforts during the evaluation process encouraged and motivated them not only trying best for the evaluation but also following our recommendations for improvement.

Follow Specific Guidelines

These guidelines provide direction for evaluators and the CMs to implement the components of the MCE successfully and ultimately produce a sound evalu- ation. Therefore, we introduced and followed the guiding principles for evalua- tors (American Evaluation Association, 2004), the program evaluation standards (JCSEE, 2011), and the MCE’s collaboration guiding principles with the CMs throughout all stages of our evaluation to ensure that all of us had a common un- derstanding of what needs to be accomplished. While the guiding principles for evaluators and program evaluation standards are essential to conducting a sound evaluation, the collaboration guiding principles are most benefi cial for evaluators using the collaborative evaluation approach. In our experience, this component of the MCE really supported us in maintaining our ethical rules in conducting the evaluation. We often explored the intended use of the evaluation results to make sure the purpose of the evaluation is formative to improve the evaluand. When clients intend to use program evaluation for other purposes, such as their personal

Utilizing Collaborative Evaluation in a Finance and Banking Corporation • 43

or political plans, not only are ethical practices not ensured, but potential confl icts may also arise due to different evaluation visions.

Similar to the interactive relationship of the six components of the MCE, all the elements of the collaboration guiding principles are related to each other. We felt that our evaluation process was really a journey of learning how to develop healthy relationships with the CMs. Everyone involved learned greatly from each other during all evaluation phases. For instance, we shared our knowledge of a high quality questionnaire and interview design and the application of classical test theory (Crocker & Algina, 1986) to writing and reviewing the items, while the CMs shared with us their knowledge of individual customers’ interests, needs, and wants in retail banking services. Day by day through evaluation activities, we and the CMs developed a mutual trust that empowered and enhanced our learning by our genuine involvement in the evaluation process with the necessary qualifi - cations for an effective collaboration. As a result of that supportive collaboration relationship and open atmosphere for discussion, we were able to obtain positive solutions for sensitive topics and complicated situations.

LESSONS LEARNED

The fundamental lesson we drew is that it is vital for evaluators to develop and maintain a good relationship and supportive environment with the CMs to reach the optimal level of collaboration during the evaluation. One characteristic that distinguishes collaborative evaluation from other evaluation approaches is the high level of involvement of stakeholders throughout the evaluation process (Ro- dríguez-Campos, 2012b). The elements of the MCE assisted us in creating active and ongoing collaboration of key stakeholders in the evaluation of Services. Par- ticularly, the CMs felt they shared the ownership of our collaborative evaluation, thanks to their active involvement. More importantly, as an evaluative organiza- tion level was achieved, our stakeholders considered evaluation an integral part of the daily activities in their programs as well as in their whole corporation.

We also found that it is crucial to have a deep understanding of all the compo- nents in the MCE to not only have a holistic view about collaborative evaluation, but also to be fully aware of the connection and interaction of these elements for the ultimate utilization of this approach. In our evaluation, we carefully followed the steps for every single element in the MCE, which is relevant to our evalua- tion context to maximize the collaborative aspect of our evaluation. In addition, specifi c examples in Rodríguez-Campos and Rincones-Gómez (2013) were very useful because they offered suggestions for solutions when we were in similar circumstances. For example, the parking lot strategy in page 114 was very help- ful for us in controlling our meeting time while still maintaining a healthy atmo- sphere with our client and CMs. Specifi cally, the CMs were high-ranked offi cials, and earlier in our evaluation, one of them tended to overtake our meetings by list- ing off her many ideas. After asking people to write down their ideas as illustrated in the parking lot strategy, the CMs found that this was helpful for them not only

44 • DIEP THI NGUYEN & THANH VINH PHAM

to focus on evaluation topics, but also to manage their thinking in a positive way. Along with the operation of the corporation, the situational characteristics sur- rounding the evaluation often change. We utilized the advantages of being a team of two external evaluators, specifi cally, took advantage of our various perspec- tives, backgrounds, and experience throughout the evaluation phases to regularly discuss and analyze the situation in order to come up with appropriate and im- mediate next steps.

Most importantly, from our point of view, the key contributing factor to our success of utilizing the collaborative evaluation approach was we truly believe in the benefi ts and advantages of collaboration. We used to work in a fi ercely com- petitive sales department for several years and were experiencing many situations when everyone was under high pressure of assigned target of sale revenue. We still had a strong belief in the power of teamwork and always performed our work in a collaborative manner. We found that some co-workers might not be coopera- tive at fi rst. However, when they recognized that we often tried to help others with our professionalism and expertise, many of them changed their attitudes. Then we could collaboratively work and together we were able to maximize our teamwork capacity. With collaboration and willingness to help always in mind, we believe that we can inspire others and enjoy working not only in a healthy but also productive environment.

REFERENCES

American Evaluation Association. (2004). American Evaluation Association guiding prin- ciples for evaluators. Retrieved from http://www.eval.org/GPTraning/GP%20Train- ing%20Final/gp.principles.pdf

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. NY: Holt, Rinehart & Winston.

Ferreira, A. F. F., Santos, P. S., Rodrigues, M. M. P., & Spahr, W. R. (2011, November). Evaluating retail banking quality service and convenience with MCDA techniques: A case study at the bank branch level. Working papers. Banco de Portugal.

Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation standards: A guide for evaluators and evaluation users. Thousand Oaks, CA: Sage.

Rodríguez-Campos, L. (2012a). Stakeholder involvement in evaluation: Three decades of the American Journal of Evaluation. Journal of MultiDisciplinary Evaluation, 8(17), 57–79.

Rodríguez-Campos, L. (2012b). Advances in collaborative evaluation. Evaluation and Program Planning, 35(4), 523–528.

Rodríguez-Campos, L. & Rincones-Gómez, R. (2013). Collaborative evaluations: Step- by-step. Stanford, CA: Stanford University Press.