Evaluation

Page 1

CO4754

CO4754 User‐Centred System Design & Evaluation

Assignment 2 – Evaluation

Website Usability Evaluation for a Large Further Education College

Page 2

CO4754

Website Usability Evaluation for a Large Further Education College

ABSTRACT The further education sector is a highly competitive environment with organizations operating on very low margins; recruitment of students is a vital activity not only to meet organizational goals, but literally for the survival of the organization. The organization website provides a cost effective platform to attract and recruit learners, including assessment of their needs and the organizations ability to using public funding to cover the course costs. The website therefore becomes a critical part of the business; the usability of the website will directly affect the recruitment numbers and the success of the business. For larger organizations delivering learning across all subject areas, at all levels from pre-GCSE to masters degree across a number of locations, this presents a challenge for the web platform. This paper will assess the usability of the recently deployed Cornwall College Group website and recommend areas for improvement.

Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation]: User Interfaces – Evaluation/methodology, Interaction styles, Screen design, User-centred design;

General Terms Design, Human Factors, Measurement.

Keywords interface evaluation; think-aloud; usability evaluation; further education;

1. INTRODUCTION The website of a Further Education College provides the first experience of the organisation for many learners. The website has many functions such as allowing online applications and enrolments, course search and comparison, as well as providing organisational information. The primary purpose of the website is sales; in this context success is getting learners to make contact with the College so that the various teams can engage with the learner and find an appropriate course for them.

It is vital that the website does not hinder the process and goes

beyond meeting the functional requirements by meeting the user needs simply. Given the broad range of students based on their age, experience and cognitive ability and the complexities of the UK Further Education system, this presents a real challenge.

This paper reports on a usability study of the primary website of removed. In particular, it reviews the literature concerning website usability assessments, describes the evaluation methods used and summarise the results including recommendations for improvements.

2. BACKGROUND Removed College Group is one of the largest Further Education Colleges in the UK delivering learning to over 20,000 learners each year. The Group encompasses three main brands, removed for anon.

The College is focused on delivering exceptional customer service; part of this strategy is the provision of self-service online tools enabling students to manage their own details. The website (removed) forms part of the self-service approach as the starting point for the learner’s journey. Following completion of a re- platforming project 12 months ago, recent internal feedback has indicated that there may be usability issues that are causing potential learners to go to other providers, hence losing the income for the College and credibility.

The College Information Systems team have been actively monitoring website usage since it was launched, primarily using Google Analytics and log file analysis. This has resulted in quick changes where it became obvious that certain pages, such as the login page, were preventing learners from submitting applications.

Since the launch of the website the College Information Systems team have developed a better understanding of the website users and their goals through the adoption of goal based design [3], it is therefore appropriate to review the website in the context of this new knowledge.

3. LITERATURE STUDY 3.1 Usability of websites Defining usability is not simple, there is a standard definition provided within ISO 9421-11 [12]:

Usability is effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments.

This definition is useful in that it refers to specific users and goals, tying together the usability definition with a goal-based design approach as described by Cooper [3]. However, the authors of much of the HCI literature have differing

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CO4754’14, Jan 24, Preston, England, UK.

Page 3

CO4754

understandings of what this means in a web context. Benhunan- Fich identified in a web context that usability can be defined as “how well and how easily a user, without formal training, can interact with an information system of a website” [1]. This definition highlights important aspects of websites that are for public consumption, usability is considered for users without formal training, the interface should be obvious and intuitive, leading the user through the site. Bernard et al. suggested that a “true usable system must be compatible not only with the characteristics of human perception and action, but, most critically, with users’ cognitive skills in communication, understanding, memory and problem solving” [2]. The highlighting of different user abilities has particular relevance to a further education college website, since the website will be accessed by users with widely varying cognitive skills ranging from pre-GCSE to masters degree students. Returning to the ISO definition, if the website is to be usable for a pre-GCSE learner and masters degree student it must satisfy them both when enabling them to achieve their goals.

Kincl and Stach [14] focus on the ISO definition pointing out that effectiveness and efficiency are necessary yet insufficient conditions for attaining user satisfaction. Lindgaard [15] states that it is not the content of the website but the emotional reaction to visual appeal. Tractinsky and Zmiri [24] point to empirical evidence that users can be more satisfied with a visually appealing website than with a website that is usable but less visually attractive. Kincl and Stach [14] apply Herzbergs’ motivation theory [9] to explain these findings with the satisfaction rating of the website being the sum of satisfaction variables, grouped following Herzberg’s theory into dissatisfiers, satisfiers and hybrids. This helps to understand that there are elements of the interface that if not present or inadequate will dissatisfy the user, those that satisfy the user and those that are neither negative nor positive but must be present.

Garrido et al [7] defined website usability in terms of the factors, accessibility, navigability, effectiveness, credibility, understandability, customisation and learnability. The definitions of the factors are very specific and therefore particularly useful within a software engineering team that do not have the benefit of a usability expert.

3.2 Usability Evaluation of Websites The effectiveness and efficiency components of usability as defined in ISO definition lend themselves to be being measurable and therefore assessed in terms of utility and ease of use [12]. Rosson and Caroll provide evidence that testing of websites needs to be based on typical scenarios and tasks[22].

There are many sources with lists of usability evaluation methods (UEM), however, Hom’s The Usability Methods Toolbox [10] provides a comprehensive review of methods with clear explanation of how to conduct the method and when its’ use is appropriate. An omission is the use of software or automated evaluation, possibly due to the age of the toolbox.

Hasan et al [8] categorise UEMs in terms of how the usability problems are identified: for example by users, evaluators or software. User-based UEMs usually involve users being observed undertaking pre-defined tasks. Evaluator-based UEMs involve having a number of expert evaluators assess the user interface to judge whether it conforms to a set of usability principles, known as “heuristics” [17]. Software-based UEMs use software tools to

identify usability problems such as Google Analytics. It involves collecting, measuring, monitoring, analysing and reporting web usage data to understand visitors’ experiences. Their study demonstrated that they have some limitations in the evaluation of the usability of web sites, relating to the fact that the web metrics indicated only a potential usability issue. They could not provide in-depth detail about specific problems that might be present on a page, this requires user testing and/or heuristic methods.

Nielsen and Molich [19] focused on evaluator-based UEM and summarized that the ideal number of evaluators required to identify 80% of usability problems was 3-5, however further studies by Hwang and Salvendy [11] demonstrated that where evaluators only have basic training in heuristics then this number should be 10±2. Hwang and Salvendy’s study was felt to be particular relevant for this study as the evaluators being used would be new to the heuristic evaluation method.

Ivory and Hearst [13] provide a detailed analysis of usability methods, highlighting that findings can vary widely when different evaluators study the same interface, even when using the same technique. The wide variation is a concern, with less than 1% overlap which they postulate implies a lack of systemacity or predictability in usability assessment. For these reasons, they concur with Dix et al [4] and recommend that several different evaluation techniques should be applied. They highlight the opportunities that the web platform provides, especially automate usability testing. Automating the testing will reduce testing costs and providing quicker feedback on alternative designs.

Tan et al [23] confirm that effective evaluation of usability requires the use of multiple methods. Their study focuses on a comparison of findings for heuristic evaluation versus user testing. Both methods were found to be equally effective in identifying different usability problems related to five categories (navigation, information content, layout organisation and structure, usability and availability of tools and common look and feel) but user testing did not identify problems relating to two issues (compatibility, and security and privacy issues).The recommendation is that the methods need to be applied at the appropriate time in the web development lifecycle, with heuristic evaluation more effective in the early stages and user testing in the later stages.

Prom [21] investigated the use of Google Analytics to evaluate and improve the design and content of web sites. The study used standard reports from Google Analytics (i.e. funnel navigation) without deriving specific metrics. Analysis of Google Analytics data enabled problems to be identified quickly and helped determine whether a site provides the necessary information to its visitors. This study has particular relevance, as Cornwall College has collected usage data using Google Analytics for a number of years; the study suggests that analysis of the data will help identify usability issues.

3.3 Usability Evaluation Feedback Since the aim of this research is to provide feedback on identified usability issues, it is appropriate that methods for providing the feedback are considered. Nørgaard and Hornbæk’s [20] study into the effectiveness of usability feedback explores how developers respond to a range of feedback formats. Their recommendation is that feedback is given through multimedia presentation, the screen dump format (aka screen snapshots [10]),

Page 4

CO4754

or redesign proposals. This study will provide feedback as redesign proposals or screen snapshots of problems.

4. RESEARCH METHODOLOGY 4.1 Selection of UEM Three UEMs were selected, heuristic evaluation, Google Analytics and CrazyEgg heatmaps, this was based on evidence that these methods are complementary in that they are able to identify usability problems from different perspectives ([8], [18], [23]). The missing UEM is user testing which is a potential significant omission but could not be completed within the timescales of this research. Google Analystics was used since it has been in use with the College for a number of years and a good amount of data exists, the application also has a wide range of features and benefits (i.e. a usable and simple interface). Crazy Egg is similar to Google Analytics, but provides more insight into exactly where users are clicking on a page and what parts of the page being view.

4.2 Research Design The design of this research is based upon:

1. Expert evaluation using Nielsen’s 10 heuristics [17]. Each evaluator had 1 hour to complete 3 tasks as described in the recently developed persona scenarios [3]:

a. Using the persona of John the School Leaver, submit an application for a public services course starting in September.

b. Using the persona of John the School Leaver research what grant funding is available.

c. Using the persona of Alice the Recreational Learner enquire about the cost of an evening intermediate Italian course at Camborne.

2. In addition to the classic method of heuristic evaluation, problems were not only recorded, screen snapshots Error! Reference source not found. were also taken to help the development team replicate, identify and resolve any problems.

3. A procedural model drawn from the work of Masemola & de Villiers [16] :

a. Set up objectives in line with research questions.

b. Determine the aspects to be measured and their metrics.

c. Formulate documents:

Initial test plan, task list, information document for participants, checklist for administrator, and determine a means of investigating satisfaction.

d. Acquire participants.

e. Conduct usability test.

f. Determine means of analysis and presentation that address the unique, as well as the usual, aspects.

g. Draw conclusions and make proposals for the way forward.

4. Software evaluation using Google Analytics followed the approach suggested by Hasan et al [8] but was limited to user flow analysis and user drop offs.

5. Software evaluation using Crazy Egg was completed following instructions on the tools website. Particular aspects that were studied were page heatmaps indicating user clicks and areas of pages that were viewed.

5. DATA COLLECTION AND ANALYSIS

5.1 Evaluators Expert evaluators were enlisted from the Information Systems team and given 30 minutes of training on heuristic evaluation before beginning. All the evaluators have significant experience in developing web based user interfaces. Although heuristic evaluation is usually performed by usability experts, the evaluators had a good fundamental understanding of web usability, and their involvement in the study would provide a valuable contribution to their personal development[5]. Hwang and Salvendy [11] predict that where evaluators have only had basic training then the number of evaluators required to identify 80% of usability problems will be 10±2 rather than 3-5 identified by Nielsen and Molich [19], for this research only seven evaluators were available.

5.2 Ethical considerations Although the College website has been implemented by a third party, members of the information systems team had been involved in elements of the design and integration. One member of the team had a lead role in the design and was therefore given a special briefing before the research was conducted to explain the objectives and approach and that the exercise was not an evaluation of their performance or skills. Whilst the team member would be present during the evaluation, they did not participate instead acting as an observer and assistant.

Neither software evaluation method collected any data that would enable the identification of individuals; therefore there were no personal data issues with these methods. However, the privacy and cookie policy of the website were checked to ensure that users were aware that tracking tools were in use for the purposes of improving user experience.

5.3 Study 5.3.1 Expert Evaluation The evaluators were given the three tasks to achieve together with a worksheet to record any violations of the heuristics. The study was completed using the live website, the evaluators were asked to follow the task to completion except to stop before submitting an actual application or enrolment to prevent the study affecting other staff.

The study took 1hr 15 minutes for all evaluators to complete. On completion of the tasks, the evaluators merged together their findings assigning a priority to each violation identified.

5.3.2 Software Evaluation Google Analytics was already in use so no configuration was required. The analysis used data for the period 11 December 2013 – 10 January 2014.

Page 5

CO4754

Crazy Egg was not installed prior to the research. Setup was similar to Google Analytics and required the insertion of a small JavaScript snippet into the website template. The analysis used data collected over a 10 day period at the beginning of January 2014. User click tracking was established on the landing page and all pages on the level below the landing page. Additionally tracking was enabled on the course search and details pages so that the users expected journey through the website was tracked.

5.4 Evaluation 5.4.1 Expert Evaluation Expert Evaluation identified 29 usability issues with the website. Consistent with studied literature there was little overlap between the issues identified by the different evaluators. Table 1 shows the number of detections of each issue and it’s assessed impact.

Table 1. Issue detection frequency and impact.

No of evaluators reporting

issue

Low Impact

Medium Impact

High Impact

1 8 13 6

2 1

4 1

The issues detected were across the range of heuristics, Table 2 summarises the number of issues violating a particular heuristic against the impact of the violation.

Table 2. Number of heuristic violations by impact.

Heuristic Low

Impact Medium Impact

High Impact

Visibility of system status 2

Maximise match between the system and the real world

2 1 1

Consistency and standards

1 4

Error prevention 1

Recognition rather than recall

1 1 2

Flexibility and efficiency of use

1 2

Aesthetics and minimalist design

1 4 1

Help users recognise, diagnose and recover from errors

1 2

Help and documentation 1

The major usability problems and possible or implemented solutions are:

 Users receive error message pages showing codes such as 400 or 500 with no indication of what they should do. The codes are not at all useful to a user, the

occurrence of the errors needs to be investigated and removed, but when an error occurs the user should get a simple message asking them to try again or contact the College using another method, such as live chat, telephone or email.

 A Christmas Closure Banner was obscuring 1/3 of the screen during the testing. This was caused by unintended use of a feature designed for significant abnormal events such as closure due to flooding or snow. The banner concept is useful but needs to be further developed so that once a user has read the message they can dismiss it for the duration of the session.

 When a user has created a basket with a combination of questions, applications and enrolments on courses, the question screen gets stuck and prevents the user from continuing. This is a bug within the web site that needs to be fixed as a matter of urgency.

 When a user asks a question about a course the validation does not provide feedback to the user if they have not completed all the required fields, it just prevents the user from proceeding. The feedback needs to be fixed.

 The course search has an auto populate predictive list that appears as the user is typing. The list that appears is not wide enough to show the full course title with the important details being hidden. Consideration needs to be given to resolving this by either increasing the size of the list or perhaps abbreviating obvious words.

All full list of findings is at Annex A.

5.4.2 Software Evaluation 5.4.2.1 Google Analytics Analysis of Visitor Flow revealed that the most popular route through the website is:

 Landing Page - /

 Search Results - /search/courses

 Course Details - /courses/

5% (363 users) iterated at least 10 times between Course Details and the Search Results before leaving the site with 40% of users leaving after the first iteration.

The next most popular flow is to the Adult Hub page and then following the search results to course details flow.

Figure 1 – Landing page screenshot two search boxes

Page 6

CO4754

The third most popular flow is of interest:

 Landing Page - /

 Site Search - /search/site

 Course Search - /search/courses

 Course Details - /courses/

Users appear to use the site search and then use the course search having not got the results they were expecting. Figure 1shows the layout of the search boxes on the landing page.

The major usability problems and possible solutions are:

 This layout should be improved by hiding the input box and displaying a site search link that when clicked expands into a search box so that the user is not presented with two areas to input data. This affect can be seen clearly on the Google home page with the recent change to hide Google Apps as shown in Figure 2.

Figure 2 - Google Apps hidden unless clicked

 The search is really the core focus of the page, the cursor could be automatically focused in the course search input box. This will also enable regular users to start typing without having to click on it [7], as well as drawing the users’ attention to the search.

 The website is used as a landing page within the College for users logging onto wireless and some users have it set as their home page. This makes understanding the Google Analytics data difficult as a segment of the initial drop off is likely to be caused by this. It is therefore recommended that wireless logon and other internal processes that redirect to the website have specific landing pages created to exclude them from future evaluations.

5.4.2.2 CrazyEgg The heatmap of user clicks in Figure 3 reveals that all of the navigational elements of the website landing page are used, however the most used parts are:

 Course Search

 Site Search

 Student Portal

 Leisure Course Hub Page

In terms of usability problems the heatmap supports the Google Analytics analysis that the site search is used regularly, probably instead of the more appropriate course search.

A combination of the hub page heat map and scroll map confirmed a usability issue that was raised through expert evaluation but was not clear within the Google Analytics evaluation.

Figure 3 - CrazyEgg heatmap of user clicks

When users click to access a hub page, such as “University” shown in Figure 4, the area of the page that is visible is just the large images. The circle representing the hub page in the navigation menu has increased in size but it is not significant enough for users to realise that anything has changed; they therefore click the navigation link again. Once the user realises that they were already on the hub page it is likely they will feel stupid[3] and it is likely that is would become a dissatisifer[14].

Page 7

CO4754

Figure 4 - Cropped Heatmap for university hub page

The visualisation of where the user is needs to be more obvious, perhaps removing the large banner image altogether from the hub page. An alternative solution could be to make the coloured horizontal bar element of the navigation menu thicker and remove the white border from the current menu so that the current page is more obvious. Both of these should be tested using Crazy Egg to see if the number of clicks to access the already selected page can be reduced. Figure 5 demonstrates a mock up of the improvements.

Figure 5 - Navigation Improvements

The heat map in Figure 4 also shows that once on the hub page users then go to the course search which they could have or should have done on the first page. Recommended improvements to the visualisation of the search many improve this, although user evaluation should be conducted to understand the behaviour and confirm if this is a usability problem. There is evidence of this problem across all hub pages.

6. CONCLUSIONS AND FUTURE RESEARCH There are clearly a small number of high impact usability problems that need remedying urgently. The evaluation has also revealed and/or confirmed a number of usability issues that whilst not preventing users from achieving their tasks, they are no doubt acting as dissatifisers such as clicking a navigation link twice because it is not clear that anything has changed.

The notable omission from this evaluation is the use of user testing. This should be done as future research as it is likely to highlight additional usability issues.

Each of the various evaluation methods used contributed something different in terms of understanding usability problems with the website. The expert evaluation probably significantly underreported the possible issues due to the inexperience of the evaluators with heuristics; however, the method is useful and

should become a regular part of the information systems team testing.

This research has only started the analysis of the available data contained with Google Analytics and Crazy Egg, more resources should be allocated to understanding how these tools can be used effectively. Additionally, resources should be allocated to investigating how current tools for automating aspects of usability testing can complement the methods used here.

It is recommended that the use of all three UEMs continue with usability evaluation becoming a key factor in all future website developments.

7. ACKNOWLEDGMENTS The research could not have been completed without the support and assistance of the College Information Systems Team.

8. REFERENCES [1] Benbunan-Fich, R. 2001. Using protocol analysis to evaluate

the usability of commercial websites. Information and Management, 39, 151-163.

[2] Bernard, P.J., Hammond, N.V., Morton, J., & Long, J.B. 1981. Consistency and compatibility in human-computer dialogue. International Journal of Man-Machines Studies, 1(1), 87-134.

[3] Cooper, A., Reimann, R. and Cronin, D. 2007. About Face 3: The Essentials of Interaction Design. John Wiley & Sons, Indianapolis, IN

[4] Dix, A., Finlay, J., Abowd, G. and Beale, R. 1998. Human- Computer Interaction (second edition). Prentice Hall, Upper Saddle River, NJ.

[5] Ebrlich, K., Butler, M.B. and Pernice, K. 1994. Getting the whole team into usability testing. IEE Software. January 1994, 89-91.

[6] Fadeyev, D. 2008, 10 Useful Techniques to Improve Your User Interface Designs. Available online: http://uxdesign.smashingmagazine.com/2008/12/15/10- useful-techniques-to-improve-your-user-interface-designs/ (accessed 11 January 2014).

[7] Garrido, A., Rossi, G. and Distante, D. 2011. Refactoring for Usability in Web Applications IEEE Software. 60-67.

[8] Hasan, L., Morris, A. and Probets, S. 2013, E-commerce websites for developing countries – a usability evaluation framework. Online Information Review. Vol 37 No 2, 231- 251

[9] Herzberg, F., Mausner, B. and Snyderman, B.B. 1959. The motivation to work. New York, NY: John Wiley and Sons.

[10] Hom, J. 1998. The Usability Methods Toolbox, available online: http://jthom.best.vwh.net/usability/usable.htm (accessed 28 December 2012).

[11] Hwang, W. and Salvendy, G. 2010. Number of People Required for Usability Evaluation: The 10±2 Rule. Communications of the ACM. May 2010, Vol 53, No5. DOI= http://doi.acm.org/10.1145/1735223.1735255.

[12] ISO 9241-11, 1998, Ergonomic requirements for office work with visual display terminals (VDTs), P11: Guidance on usability.

Page 8

CO4754

[13] Ivory, M.Y and Hearst, M.A. 2001. The State of the Art in Automating Usability Evaluation of User Interfaces. ACM Computing Surveys. Vol 33, No 4, 470-516.

[14] Kincl, T. and Strach, P. 2012. Measuring website quality: asymmetric effect of user satisfaction. Behaviour & Information Technology. Vol 31, No 7, 647-657.

[15] Lindgaard, G. 2007 Aesthetics, visual appeal, usability and user satisfaction: what do the user’s eyes tell the user’s brain? Australian Journal of Emerging Technologies and Society, 5 (1), 1-14.

[16] Masemola, S.S. and de Villiers, M.R. 2006. Towards a Framework for Usability Testing of Interactive E-Learning Applications in Cognitive Domains, Illustrated by a Case Student. In J. Bishop and D. Kourie, Service-Oriented Software and Systems. Proceedings of the 2006 annual research conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT) 2006:187-197.

[17] Nielsen, J. 1994. Usability Inspection Methods. Wiley.

[18] Nielsen, J. 2003. Usability 101: introduction to usability. available at: http://www.nngroup.com/articles/usability-101- introduction-to-usability/ (accessed 11 January 2014).

[19] Nielsen, J. and Molich, R. 1990, HE of user interface. In CHI ’90 Conference Proceedings. ACM, 249-256.

[20] Nørgaard, M. and Hornbæk, K. 2009. Exploring the Value of Usability Feedback Formats. International Journal of Human–ComputerInteraction. 25(1), 49-74.

[21] Prom, C. 2007. Understanding on-line archival use through web analytics, ICA-SUV Seminar, Dundee, available at: www.library.uiuc.edu/archives/workpap/PromSUV2007.pdf( accessed 11 January 2014).

[22] Rosson, M.B. and Carroll, J.M. 2001. Usability engineering: scenario-bsaed development of human-computer interaction. San Franciso, CA: Morgan Kaufman.

[23] Tan, W., Dahai, L. ad Bishu, R. 2009. Web evaluation: Heuristic evaluation vs user testing. International Journal of Industrial Ergonomics. Vol 39. 621-627.

[24] Tractinsky, N. and Zmiri, D. 2006. Exploring attributes of skins as potential antecedents of emotion in HCI. In:P. Dishwick, ed. Aesthetic computing. Cambridge: MIT Press, 405-422.