Project Assignment (1500 words)

Project Assignment 03

This Assignment is concerned mainly with evaluating the redesigned system that you

submitted for Question 4 of Assignment 02. Please make sure you include a copy of question

4 of Assignment 02 with this assignment. Similarly, if you have made any changes to your

design in Assignment 02 as a consequence of the graders' comments, then please include a

copy of the new design. The purpose of this is to ensure that the graders have a copy of the

UI you are evaluating – otherwise, they will have difficulty grading this assignment.

Question 1: Observing the users - [about 1500

words]

User-observations are explained in Chapter 20, 21 and are discussed throughout part 4.

1.1 (10 pts) Write your evaluation strategy for observing the users interacting with your redesigned user interface.

1.2 (15 pts) Write an evaluation plan. In particular:

1. justify your approach to user observation with reference to the elements of evaluation

2. justify your choice of participant(s), and how they match the user groups/personas you created earlier in this project

3. justify the tasks to be carried out by the participants 4. justify the location of the evaluation session 5. explain how you anticipate running the session and collecting the data

Now observe at least two users, before attempting the rest of the assignment.

1.3 (15 pts) Describe how you carried out your evaluation, comparing what actually happened with what you planned. Check the rubric for more details on our expectations.

Rubric

This criterion is linked to a Learning OutcomeQuestion 1.1

The evaluation strategy is dealt with in Chapter 21. Make sure you stick to user observation

techniques as described in Chapter 20; inspections and standards are not acceptable here.

The following questions need to be considered: • Why am I evaluating? You have to provide

justified, clear reasoning and have considered the alternatives. • Which usability requirements

am I exploring? This should follow from the rationale for the re-design from Assignment 02

and should introduce ideas of the difference between qualitative and quantitative data. The

student may refer back to the metrics previously identified. There is a lot of discussion of

metrics and appropriateness in Chapter 21. Clarity of aim and appropriateness to the re-

design and the stage of the prototype are more important than introducing a lot of detailed

bookwork. • What type of data do I want to collect? This must discuss qualitative vs.

quantitative considerations, justified not only by practical constraints – which are addressed

below – but also with reference to what is appropriate for the aim and the requirements being

explored. Timing tasks, identifying errors. • What am I evaluating? This should describe the

kind of artifact, low-fidelity vs. high-fidelity, e.g. semi-working prototype, (say, by taking notes

and videoing). Working scenario into storyboards. • What constraints are there? Constraints

due to the quality of prototype, time, money, etc. Two points each.

This criterion is linked to a Learning OutcomeQuestion 1.2

The evaluation plan (see Chapter 22) – should follow logically from the strategy. Some

questions only ask for justification, but we expect descriptions of users, tasks, etc in the

appropriate places in order to justify them. The answer should follow the form of Chapter 22

developing the evaluation plan document so that by the end you should understand what is

being done and why. • Justification of approach and technique. For explicit mention of

elements of evaluation: compare, observe, listen, and measure. [1]. For a description of

techniques to be used and data to be obtained. [1]. For justification in HCI terms, including

complementary data between techniques (not purely practical constraints – see Table 27.4).

Also, see Chapter 27 for a discussion of complementary techniques. [1] • Justification of

number and choice of participants: mention real users, experienced users of the old system,

especially those who experienced problems the re-design is intended to address,

representative user, HCI or domain expert and explain why they have been/have not been

used. (Table 22.1 and Table 22.2). [3] • Justification of tasks – are they typical tasks, important

to the redesign, etc? Refer to the criteria in Chapter 22, including Table 22.8. This must follow

the problems identified in Assignment 01 and the re-design in Assignment 02. [3] • Location

(Chapter 26) – discuss relevance to the actual environment, noise, disturbance, show that you

know the difference between field studies and controlled studies, discuss constraints [2] •

Planning the session: scheduling, equipment, pre and post questionnaires and other

evaluation material, piloting (see Chapter 22). Actual materials can be included as appendices,

require justification for full points. [4]

This criterion is linked to a Learning OutcomeQuestion 1.3

Description of user observation sessions. This question should not include data, that comes

in question 3: it is intended to provide evidence that you did actually do the observation. How

well did the techniques used to provide the predicted information? General points that may

be mentioned include: • How many users didn’t turn up as promised? Did this unbalance the

experiments or merely reduce them? • Was it impossible to complete the tasks? If so, look for

explanations as to why, e.g., lack of time or because the users couldn’t drive the system, or

some other reason. • Any problems with the location. • Any problems with evaluation

materials, scheduling, etc. Think aloud tape is barely audible or silent, video on screen only,

and does not capture user expression, equipment failure, etc.