Usability Engineering
meerabsheikh
UE2020_week4-3.pptx
Usability engineering How to conduct User testing
Week 4
How to conduct User Testing?
In Week 1, we talked about why users are important for computer systems and discussed What is Usability?
In Week 2, we discussed the fact that users are all very different and the need for having a test plan
Last week, we explained how to design a plan for user testing.
This week, we will explain more about how to actually conduct usability testing and what things we can test …
The $300 million button
Developers thought
that this button would …
enable repeat customers to purchase faster
first-time customers to register because it would make future purchases faster
Web Form Design: Filling in the blanks, by Luke Wroblewski (2008) Rosenfeld Media, http://www.uie.com/articles/three_hund_million_button/
Register
Login
Forgot Password?
Add items to cart
Login / Register
Purchase Items
Step 1
Step 2
Step 3
However, The usability test found …
First timers
Couldn’t remember if it was their first time
Made multiple false attempts at login
Felt the retailer was going to use their information to pester them
Repeat customers
Couldn’t remember which email/password they’d used
Used the Forgot Password link if they could remember which email
“I'm not here to enter into a relationship. I just want to buy something.”
You do not need to create an account to make purchases on our site. Simply click Continue to proceed to checkout. To make your future purchases are even faster, you can create an account during checkout.
Continue
Login
Forgot Password?
Note: This is a representation of changes. Actual screenshots aren’t available because Spool did not reveal the company.
The solution…
This lecture Content
Before the test
Pre- & post-questionnaires
Conducting the Usability Test
Roles: facilitator, observer and user
How to record observations
Report Writing
From observations to recommendations
Common Industry Report template
Recruiting tips for user participants
Avoid power users
They skew results
If recruiting within
your company …
Don’t let users’ managers observe
Follow up with schedule
Send a clear and detailed confirmation; Make a confirmation call
Screener questions
Typically 20 questions
Clear and specific, no jargon, exact dates, quantities, times
Questions should not lead
“Are you bothered by ….”
Every question should have a purpose
Start with questions that screen out the most people.
Screen participants
Give reason for the session
Feedback to help us improve the product
Not a sales call
Give length, date, location of session
Offer incentives up front
Explain video/audio taping, if it will occur
Go through screening questions if person is interested
Having decided on the users to test ……
Before the test…
24 hours before, check the following for yourself …
Tasks can be completed
Tasks are clear and understandable
Materials are available
Time allotted is appropriate
Number of tasks is appropriate
Nothing is overlooked: bugs, goof-ups
If system is under development, check state of application
Pre- and post-test questionnaires
Pre-test questionnaire
Some of your screener questions
Confirm that a participant fits a user profile
Locates a participant with user profile ranges
Include other characteristics / behaviour / attitudes that might be relevant to the Web site or tasks you are testing
Web access from home / work / public computers
Usage of competing / similar Web site
How they prioritise their Web time
Post-test questionnaire
Find out how a participant feels about the site/software being tested
Complements your measurements of efficiency and effectiveness
Based on a standardised opinion questionnaire with a scoring system, plus open-ended questions
Features you would like to see / do not like
Produces a list of satisfying and unsatisfying site/software features – especially useful if testing is done during development
Perform Usability test
How to measure it?
Where to perform it?
System usability scale (SUS)
I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
Brooke, J. (1996). SUS-A quick and dirty usability scale. In P. W. Jordan, B. Thomas, McClelland, & B. Weerdmeester (Eds.), Usability evaluation in industry (pp. 189–194). London, UK: Taylor & Francis.
System usability scale (SUS)
Likert scale
Odd questions
Subtract 1 from score
Even questions
Subtract score from 5
Add scores, multiply by 2.5
disagree agree
1 2 3 4 5
System usability scale (SUS)
Sauro points out
SUS Scores are not percentages
The average score is 68
http://www.measuringusability.com/sus.php
Conducting usability tests
Formal usability lab
Facility design
Low-traffic low-noise area or good sound insulation in walls
Good acoustics and lighting
Electrical, ventilation & ceiling requirements
Double-pane two-way mirror
System design
Digital audio/video/screen-activity monitor/capture
Event-logging software
Editing system
Formal usability lab
Advantages
Provides a dedicated testing area
Shows that management is supportive
The technology impresses the developers
Considerations
It’s expensive
Creates a bad image if not used regularly
Can be intimidating for users
Testing at the users’ site
When looking for contextual information:
How do people work in their environment?
What tools do people use?
What is their work flow?
If people use devices you don’t have in the lab:
Highly-customized interfaces
Assistive technology: screen magnifier / reader
Unconventional work environment
When users are difficult to schedule
Logistical considerations
| Onsite | Usability Lab | |
| Advantages | No lab set-up or installation to do You test in the real situation | Efficient Users travel, not you or your team More people can observe You can control the environment |
| Challenges | Setting up the visits so they run smoothly Capturing information Is there a fast Internet connection there? Can't install Camtasia etc there Harder to take screen shots | Setting up or booking the lab |
Screen-recording software
Camtasia [30-day trial]
General screen activity and audio recording
Mainly used to record instructions / demos
Can use video editing software to inset Webcam video of user
Morae [web demos]
Special-purpose usability software
Automatically insets video of user in screen video
Supports all stages of usability testing
HyperCam [V2 free]
Record Windows screen and sound
CamStudio [Open source]
Silverback [Mac, V2 free]
Testing materials
Screening questionnaire
Orientation script
Background questionnaire
Data collection instruments (Data Loggers)
Nondisclosure Agreement and Tape Consent Form
Pre-test questionnaire
Task scenarios
Prerequisite training materials (optional)
Post-test questionnaire
Debriefing topics guide
Testing procedure
Greeting/Consent Form/ Non-disclosure agreement
Practising thinking aloud (familiarising with interface)
Task 1
Task N
Post Questionnaire / Interview
Debriefing / Concluding
Entry
questionnaire
.......
Encouraging thinking aloud
Facilitator (Moderator)
Greet and debrief
Greet and debrief user
Offer drink and explain test procedures
Get
Get the user to sign consent forms
questionnaires
Ask clarification questions
Give gift /cash
Get
Get users to interact with interface
Encourage think aloud
Ask appropriate questions to seek clarification
Coaches the user to think aloud by asking questions
Observe and take
Observe and take notes (active listening)
Control
Control the session prevent observer interruptions
Asks for questions from observers towards the end
Written copy of the task?
For
If task is asking participant to find something with a difficult spelling it would be better to give instruction on a task card. Allows them to reflect back/re-read the task.
Get them in the mood.
Cognitive load increase by not giving them written task.
Against
if using a physical object it’s better to engage in a form of role play to make the task more like how it would be done in the real world
What if they are vision-impaired?
Dumas (p177)
Facilitating - What to do
Acknowledge the unnatural situation
Remain
neutral
Don’t give any clues as to where the user should go next
Don’t ask leading questions
If the user doesn’t use a feature you’re interested in:
“Could you go back and try it this way?”
Give subtle acknowledgements
“Uh huh”, “ok”, nod
Don’t enthuse when they find the answer
Avoid explaining or defending the interface
Avoid answering questions or helping the user too early
Don’t show the user how to complete the task, except in the debriefing at the end of the session
Facilitating - What to do
Don't ask "how was your day? - a pseudo-relationship skews the test
Let the user explore for 5-10 minutes - then give the first specific task
Take notes even when nothing is happening - If you stop writing, the user will stop talking
Listen actively and be aware of the user's body language
If prototype is unstable -Tell user not to worry, you will guide them through parts that don’t work
Minimise distractions - Hide equipment, clear the desktop
Manage observers
During the test
Focus on user's immediate experience, not extrapolation:
Don't ask users to predict the future
Non-judgmental
Focus on a single topic
Open-ended, not binary
What would you do … if …?
The user is uncomfortable being video/audio recorded
The user is uncomfortable signing the non-disclosure forms
The user starts to blame her/himself
The user doesn’t fit the right profile
Observing - what to do
Be quiet
Observers take notes, and give these to the facilitator after each session
As facilitator, you want them to have something to do
They feel involved
You get more data
Observers may participate in debriefing the users
User – What to do
The user
Signs consent forms
Completes pre & post Questionnaires
Interacts with the interface
Attempts to complete tasks
Thinks aloud
Think-aloud protocol
Preparation
Explain the difference between explaining and thinking aloud
Emphasise the need for continuous verbalisation
Practice! Given an example
Not perfect
Feels weird
Thinking aloud may alter behaviour
Disrupts concentration
Think-aloud alternatives
Pairs of users
Working together people will discuss and talk
Disadvantages
Pairs will talk less if they know each other
Twice as expensive
Retrospective Think-Aloud
Review and discuss video with the user
Allows you to record ‘time on task’
Useful in eye-tracking studies
Recording observations
Pen & paper notes
Prepared forms can help
Audio recording
For think-aloud
Video
recording
Often two cameras, one for user’s face, one for screen
Good for closed-circuit view by observers in another room
Generates much data
Screen recording software
Software logging
Eye tracker
Using pen & paper
Get to know the site really well
Try to find different ways to do the tasks
Do a trial user test
What measurable outcomes can you record?
#clicks, order of choices, etc.
Develop shorthand symbols that you will use
Identify pages, menus, page areas, links, etc.
Identify actions: ^ = click, b = back, ~ = scan, etc.
Idea
Print screen shots (perhaps several per page) to mark the path the user takes
Tips for taking notes
Create completion codes
| SC | CX | GU | ST |
| Successfully Completed | Completed, wrong answer or guessed answer | Did not complete task (Gave Up) | Task stopped by facilitator |
These examples were provided by Scott Rippon
Keys for note taking
Observation key
(!): Issue
“”: Quote
(P): Facilitator prompted
(Q): Participant question
(IDEA): Design idea
Recording user path
└> Link name: Clicked a link
Blah: Filled in text box
[Button]: Clicked a button
[✓]: Checked checkbox
(●): Selected radio button
These examples were provided by Scott Rippon
From observations to recommendations
Transcribing notes
Different ways of recording notes
Sequential observations (by time)
Global observations (by task)
Sequential observations
By time
| Time | Page Name | Observations | Comments | Issues (write this later) |
| 08:07:19 | X | Select an item | “this looks interesting” | |
| 08:09:20 | X | Put the item into shopping cart | ||
| 08:10:00 | X | Click “check out” | ||
| 08:10:56 | Y | Remove an item | Error “Why can’t I simply remove this item? Oh, no, I have to restart the whole process again” |
Global observations (by task)
Task 2 – Join a network
You would like to join a group of professionals in your area.
Task Discussion questions:
Is this something you would be professionally interested in doing?
What are some of the reason/things you would like to do by joining this group?
Task Answer:
ACME Online > ACME Networks > Browse/Search for group > Join Group
How would you go about doing this?
Global observations (by task)
After the usability sessions transcribe your observations into a spread sheet
| Task | Pilot Participant | Participant 1 | Participant 2 |
| 2 | Tried to find groups in global nav options. “Doesn’t come up.” Looked in Networks and Resources > Links to find. | Looked in ‘Networks’ > ‘Upcoming Network Meetings’ first. “Try that” “Browse through these” Would send email to RSVP address to ask to join group. | Looked in Networks menu then clicked the Find a Network Meeting link on home page. Networks listing: “Not intuitive list” “Don’t seem to be sorted.” “Don’t get it.” “Feel a bit flat” |
| 3 | Would contact ACME and ask to have details forwarded on. Looked in ‘About ACME’, ‘Networks’, ‘Resources’ and ‘Contact’. Did not think to sign in and look in members area. Idea: Could link to ‘Find a mental health professional’ on Contacts page. | Speed: “Very slow” | Member listing: “This alphabetical order is f****d!” Advanced search link: Accidentally clicked ‘Advanced search’ link to submit search. |
Collating task completion results
Group issues into themes
How scores & themes contribute
Report writing
Purposes of reports
A report makes findings understandable and actionable
Presents issues and usable designs in a simple format
Has necessary and important information
Is available in a timely manner
Should include quotes; could embed video clips
A report is a historical document
A report can be impressive: a fat consulting document
Be succinct
“Brevity is the sole of wit” – Shakespeare
“Omit needless words” – Strunk
All writers will have to edit their prose, but the great writers edit it viciously, always trying to eliminate words which are ‘fuzz’ – excess words which are not adding anything of value. Zinsser compares the process of editing out ‘fuzz’ to fighting weeds – you will always be slightly behind because they creep in when you aren’t looking for them. One of my pet hates is the word ‘also’. If you search and replace all instances of this word you will find you can live without it and your writing will improve instantly. Likewise the word ’very’.
De-clutter your writing
All writers edit their prose, but great writers edit viciously. The point of editing is to eliminate ‘fuzz’, or excess words which don’t add value. Zinsser compares removing ‘fuzz’ to fighting weeds – you will always be slightly behind. Two examples of fuzz are ‘also’ and ‘very’. Work at keeping them out of your text and your writing will improve.
Example from Ingur Mewburn (RMIT) http://thethesiswhisperer.wordpress.com/2010/11/04/5-ways-to-declutter-your-writing a book review of ‘On writing well: the classic guide to writing non fiction’ by William Zinsser
Who is your audience?
In this course:
Lecturer and tutors, other students who may take this subject in future (only if you give permission), maybe future employers
In industry your clients may be:
Engineers
Designers
Project managers
Principal investigators (other researchers)
Knowing your audience
Who is getting (this version of) the report?
What are their immediate/long term goals?
What do they know?
What do they need to know?
What are they expecting to get from you?
Common industry format
Title Page
Executive Summary
Introduction
Method
Results
Appendices
Common industry format
Title Page
Executive Summary
Introduction
Method
Results
Appendices
Product (and version, if necessary) tested
Test: who led it, and when
Report: date, author, and author contact info
Customer company and contact person
Common industry format
Title Page
Executive Summary
Introduction
Method
Results
Appendices
Name and brief description of the product
Brief summary of method(s) including
number(s) and type(s) of participants
and tasks.
Reason for and nature of the test.
Summary of results
Common industry format
Title Page
Executive Summary
Introduction
Method
Results
Appendices
Full product description
Context of use (e.g. users, tasks, equipment, environment)
Test objectives
Common industry format
Title Page
Executive Summary
Introduction
Method
Results
Appendices
Participants: who did we work with?
Context: what tasks tested, where, & when
Experimental design: how was it tested? How are data collected?
Metrics: how did we evaluate effectiveness/efficiency/satisfaction?
Common industry format
Title Page
Executive Summary
Introduction
Method
Results
Appendices
What did we learn?
Tables, charts, photographs
Perhaps recommendations
Common industry format
Title Page
Executive Summary
Introduction
Method
Results
Appendices
Full text of questionnaires
Interview protocols
Extra detail on research context, if necessary
User Quotes
Journalists looking for car photos
“You get the feeling that some 20-year-old was given all this money and they were told to make it slick … If you’re having a mid-life crisis, this is the place to go.”
“It looks like a bunch of dog dishes” [overlapping BMW wheel covers]
Creating a convincing report
Acknowledge limitations
What are the limitations of the test design?
What are the data collection problems?
What are the limitations of the analysis?
Especially quantitative analysis
Communicating results
Involve and acknowledge all team members
Ask for their input
Foster inclusiveness in presentation
Don’t have surprises
Avoid opinions; stick to facts
Be positive and proactive
Let’s look at an example
Common usability problem
Three different ways of wording the recommendation
Writing recommendations
Problem: Credit card images: the icons in the right hand side (above the CardHolder field) appear clickable, but they are just meant to indicate what cards are acceptable.
From: Molich, Jeffries and Dumas (2007). Making Usability Recommendations Useful and Usable. Journal of Usability Studies Vol. 2, pp. 162-179
Recommendation #1
From: Molich, Jeffries and Dumas (2007). Making Usability Recommendations Useful and Usable. Journal of Usability Studies Vol. 2, pp. 162-179
“Checkout: credit card icons look clickable.
Some users may be inclined to click on the credit card icons to specify which card they are using.
Suggested solution: Change the visual presentation to discourage this unnecessary behaviour.
Usefulness rating 5.0, Usability rating 1.0
This advice is vague. Several teams were vague about how the appearance of a selected icon would be different from a non-selected one.”
Recommendation #2
From: Molich, Jeffries and Dumas (2007). Making Usability Recommendations Useful and Usable. Journal of Usability Studies Vol. 2, pp. 162-179
“Credit card icons cause users to feel they must select one for a successful transaction.
Several users felt that they had to click on the proper credit card icon for a successful reservation. When nothing happened upon clicking the icon, users assumed that something was broken. One user actually typed the credit card name in the ‘CardHolder’ text input box because she felt that was what clicking on a CC icon should do (she thought it was just broken).
Suggestion: The icons appear to serve no purpose. If this is the case, they should be removed so as to avoid any confusion.
Usefulness rating 1.3, Usability rating 5.0
The above assumption is incorrect. The icons inform users of which credit cards are accepted by the hotel. On the other hand, the recommendation (‘remove icons’) is very precise and actionable.”
Recommendation #3
From: Molich, Jeffries and Dumas (2007). Making Usability Recommendations Useful and Usable. Journal of Usability Studies Vol. 2, pp. 162-179
“Form – credit card icons
Credit card icons are not clickable, but most people tried to set card. People are used to providing the card type along with the number and expiration date. It is not widely known that card type is redundant.
Suggestion: To make this display fit the mental model of the user, it would be good if the icon reacted like buttons. These “buttons” need not work on the back side but would help the user.
Usefulness rating 5.0, Usability rating 4.3”
Presentation ideas
Creative visualization makes data memorable
Techniques
Quotes from report
Annotated screenshots
Videos (brief)
Photographs
Charts and graphs
References
Kuniavsky, M. (2003), “Creating Tasks”, Observing the User Experience, Morgan Kaufmann, San Francisco. Pages 270-275.
Dumas, J., Redish, J. (1999), “12. Creating task scenarios”, A Practical Guide to Usability Testing, Intellect Ltd., Exeter. Pages 171-182.
