case study 1
Making the Case for Quality
Six Sigma Optimization of Mystery Shopping: A Hypothetical Case Study on the Nigerian Banking Industry
• Mystery shopping (MS) can be a very valuable exercise for studying and evaluating service delivery performance within the banking industry.
• Using Six Sigma tools and hypothetical data, this case study tests the approach and results to gauge poor service from excellent service delivery.
• The MS approach is highly applicable as a balanced scorecard parameter to measure delivery within service centers.
At a Glance . . . Mystery shopping (MS) is
a tool used externally or
internally to measure the
quality of service and com-
pliance with standards and
regulations. It can increase
employee awareness about
the customer to improve cus-
tomer satisfaction and deliver
excellent service, which
in turn increases sales and
reduces risk exposure.
In the MS process, param-
eters such as staff attitude,
knowledge index, turnaround
time, and empathy are tested
to monitor service delivery
performance. A quality defect
is the difference between
expected and observed variants. Operational opportunities that could cause defects are the total number
of service operators tested by mystery shoppers.
MS metrics have often been used directly in bank branch scorecards. With the use of defects per
million opportunities (DPMO) and sigma level, the metrics can be enhanced. The level of statisti-
cal robustness is further improved with the application of rolled throughput yield (RTY) to optimize
quarter-on-quarter MS results.
This case study presents hypothetical data to test Six Sigma process metrics in MS. The results show
great variance between poor and excellent service delivery in accordance with sigma level. The meth-
odology used in this case study can be applied in any service-rendering organizations that intend to use
the MS approach in measuring service delivery performance.
by Tewogbade Shakir
September 2014
ASQ www.asq.org Page 1 of 4
Background
Most Nigerian banks strengthened their capital base during the consolidation era from 2005 to 2006. With consolidation, expec- tations on the part of the banks increased, with benefits from synergies, human and material resources, and information tech- nology. Expectations from the service delivery perspective have further increased competition among banks. Customers now seek excellent service delivery to make deposits, process transfers, withdraw funds, use debit/credit cards, open accounts, and apply for loans. To enhance awareness among branch personnel and reinvigorate performance, many banks have introduced the MS tool. This is used to track both emotional and technical attributes exhibited by service operators in delivering excellent service to customers. MS metrics address attributes ranging from greeting, eye contact, and attention to product knowledge. The metrics are recorded on a monthly basis, while they are used quarter-on- quarter in a financial year.
The mystery shopper does not test equal numbers of processors (staff who serve customers by carrying out one or more process- ing functions) in various branches. Direct measurement does not factor this in. Six Sigma processor opportunities create a better measurement; the more processors tested by a mystery shop- per, the greater opportunity there is of getting it right. Also, the sigma concept allows measuring the progress or regress of ser- vice delivery from one quarter to another with rolled throughput yield (RTY).
Perhaps the methodological application of Six Sigma can also be utilized in other service-rendering organizations in the same manner, using general attributes (technical and emotional) out- lined in this case study.
Methodology
Many Six Sigma metrics are based on understanding defects in features or attributes in product, process flow, or service deliv- ery. Defect rates can be used to track quality performance. In the MS exercise, defects per operator opportunities will determine if a service outlet is delivering service to a high level with ease. The following metrics will be considered in this paper:
• Defects per million opportunities (DPMO) • Rolled throughput yield (RTY), quarter-on-quarter • Sigma level
DPMO
DPMO is a measure of the number of defects occurring in a business process. It simply expresses how process flow, service, or product is performing in relation to quality defects. DPMO is the total number of defects in a population divided by the total
number of opportunities, multiplied by a factor of 1 million. Explicitly, it shows the probability of items (attributes) with zero defects, where opportunities for defects can vary. In this MS example, the number of processors contacted yields the total number of defect opportunities:
Opportunities = Total number of service processors queried by MS
The opportunities factor normalizes the varying number of ser- vice processors contacted in different service outlets. It indicates the chance of a processor proffering positive attributes to avoid faults (defects).
DPMO = 1,000,000 × weight of attributes with fault
Total weight of attributes × no. of opportunities
Typical technical and emotional attributes tested in an MS exercise include:
• Greeting • Courtesy, politeness, and empathy • Correct response the first time • Understand MS question • Answer question correctly • Clear understanding of service procedures • Efficient handling of MS transactions • Referral to appropriate officer • Turnaround time • Eye contact and smile • Confidentiality • Boldness and confidence • Environment and ambiance • Product knowledge • Appearance
Consideration is made for attribute weights, and the weight rate is often decided by management or the MS project owner in accordance with business/enterprise objectives. Weight distribu- tion can be linear or nonlinear. The opportunities treatment will be the same for both distributions.
Processors’ responses are rated from bad to excellent, using a scale of 0 to 10 points. Mystery shopping is a subjective approach and such rating allows attributes to be scalable. The aim of the wide gap is to easily separate bad service delivery from excellent attributes. Tables 1 and 2 present the points and weights used for the purposes of this case study.
Rolled Throughput Yield (RTY)
Financial years include 12 months, which are segmented into four quarters. Since human behavior is nonme- chanical and varies with time, DPMO attributes will vary
ASQ www.asq.org Page 2 of 4
quarter-on-quarter. This effect is normalized using RTY; thus, MS results and service performance are monitored as the financial year progresses.
RTY will measure overall quality level of Qn+1 on Qn for each service outlet:
RTY = [1 – DPMOQn ] × [1 – DPMOQn+1]1,000,000 1,000,000 Sigma Level
Sigma is a measure of degree of variation in a process flow. Six Sigma focuses on defect prevention; thus in the MS project, as a faulty response is reduced, sigma level is increased. There is an inverse relationship between sigma level and DPMO. As DPMO decreases, sigma level increases and quality of the processors’ responses will be high. Thus, service processors deliver excellent service when all customers are treated like mystery shoppers. Sigma focuses on results that are critical to customers’ satisfac- tion. See Table 3.
Analysis and Interpretation
Hypothetical data in an MS project for three service outlets are presented in Table 4.
DPMO is calculated for each branch, with the following opportu- nities (see Table 5):
Igan Branch: Opportunities = 3 Akintaro Branch: Opportunities = 8 Imashahi Branch: Opportunities = 2
Table 4 — MS data for three service outlets
Branches
Igan Akintaro Imashahi
Attributes Q1 Q2 Q1 Q2 Q1 Q2
Greeting Fair Poor Excellent Excellent Bad Poor
Courtesy, politeness, and empathy
Poor Poor Excellent Excellent Bad Poor
Product knowledge
Good Poor Excellent Excellent Poor Bad
Service procedure and efficiency
Fair Fair Excellent Excellent Poor Bad
Referral to appropriate officer
Poor Fair Excellent Good Fair Poor
Turnaround time
Good Fair Excellent Excellent Poor Fair
Eye contact and smile
Good Good Good Excellent Bad Fair
Confidentiality Fair Good Excellent Excellent Fair Fair
Environment and ambiance
Fair Fair Excellent Excellent Fair Bad
Appearance and confidence
Fair Good Excellent Excellent Poor Poor
Total weighted points
1,150 940 1,960 1,980 450 400
Table 3 — Interpreting sigma levels
Sigma Level DPMO % Defects % Success Capability Cp
1 691,462 69 31 0.33
2 308,538 31 69 0.67
3 66,807 6.7 93.3 1.00
4 6,210 0.62 99.38 1.33
5 233 0.023 99.977 1.67
6 3.4 0.00034 99.999966 2.00
ASQ www.asq.org Page 3 of 4
Table 2 — Point weighting scale
Response Weight
Excellent 10
Good 8
Fair 5
Poor 2
Bad 0
Table 1 — Points allocated to each attribute
Attributes Points
1. Greeting 10
2. Courtesy, politeness, and empathy
20
3. Product knowledge 40
4. Service procedures and efficiency
30
5. Referral to appropriate officer 10
6. Turnaround time 20
7. Eye contact and smile 20
8. Confidentiality 20
9. Environment and ambiance 20
10. Appearance and confidence 10
Table 5 — DPMO and RTY values
DPMO (Q1) DPMO (Q2) RTY After Q2
Igan 141,666 176,666 0.71
Akintaro 2,500 1,250 1.00
Imashahi 387,500 400,000 0.37
The RTY after the second quarter implies a 71 percent success rate for the Igan service outlet, 99 percent success for Akintaro, and 37 percent success for Imashahi. Reject/defect quality is a categorical factor in Six Sigma projects; thus, chi square meth- ods are used to study reject distribution of the three branches. This is certainly applicable for the MS study project where service outlets are classified along directorate or area offices. Assuming the Igan, Akintaro, and Imashahi branches are the only three branches within the Yewa area office, then the defect distribution is shown in Table 6.
Testing the critical value at α = 0.10, while the degrees of freedom are (R-1)(C-1) = 18, the critical value of chi square, χ2 = 501. This exceeds the critical value, suggesting that branches differ with regard to proportions of various types of defects in this particular business development office. The resultant yield for area offices can be computed, making use of effective sum of defects encountered and total number of proces- sors tested. It is usual for the reporting area office/directorate to have an independent scorecard composed of resultant data from service outlets under such directorate in big enterprises (Table 7).
Table 7 — Reporting area office scorecard
Total Defect for Q1
DPMO for Q1
Total Defect for Q2
DPMO for Q2
RTY After Q2
Yewa Area Office 2,440 31,282 2,680 34,359 0.94
This area office had 93 percent success in the MS exercise after the second quarter. The success rate is relatively high because the branch with excellent service delivery attributes had the larg- est number of opportunities (Akintaro branch: 8).
Results and Remarks
MS project results are utilized as a metric in the branch’s scorecard. The scale gauging and usage categorization is
appropriately computed by the business/project owner. The sigma levels for the selected branches are 2.08, 4.89, and 1.16, respectively. The scale is chosen after running empirical tests over disparate MS data. The robustness lies in defect opportuni- ties created by total number of processors interviewed by MS. RTY is used to measure overall quality from one financial quar- ter to another.
Similarly, area office performance can be calculated with RTY using the DPMOs of all branches under its directorate.
The MS exercise emphatically shows data that can be used to proactively prevent subsequent defects. It is a continuous improvement tool in service outlets. Identified defects will also show learning curves for future operation and processing. The main focus for any service provider is to have customers come back again and again. MS is a great systematic approach to improve service delivery to service users (customers).
For More Information
• To contact the author of this case study, email Tewogbade Shakir at pingcommercial@gmail.com.
• To read more examples of quality success, visit the ASQ Knowledge Center at asq.org/knowledge-center/case-studies.
About the Author
Tewogbade Shakir works in the financial industry in Nigeria. He has a bachelor of science degree in physics from the University of Lagos, Nigeria. He is currently working on the thesis for his master’s degree in information technology. A member of ASQ and the Project Management Institute, Shakir is a Six Sigma Yellow Belt. He is an ASQ Certified Quality Improvement Associate (CQIA) and a PMI Certified Associate in Project Management.
ASQ www.asq.org Page 4 of 4
Table 6 — Defect distribution for MS service outlets
Yewa Greeting Empathy Product Procedures Referral TAT Eye Contact Confidentiality Ambiance Appearance Total
Igan 50 160 80 150 80 40 40 100 100 50 850
Akintaro 0 0 0 0 0 0 40 0 0 0 40
Imashahi 100 200 320 240 50 160 200 100 100 80 1,550
Total 150 360 400 390 130 200 280 200 200 130 2,440