web development and usability

Old PAWS Usability Test 1

Running head: OLD PAWS USABILITY TEST

Usability Test Document of Frostburg State University’s Old PAWS Interface

Michael B. Flinn, D.Sc.

Old PAWS Usability Test 2

Introduction

Frostburg State University is one of 11 universities in the University of Maryland

system. Frostburg State University is a small college located in the Appalachian

Mountains in western Maryland. At the beginning of the fall semester 2006, there were

4,321 undergraduate students, 720 graduate students, and 351 faculty members. The

current tuition is $6,392 for a Maryland resident and $15,442 for a nonresident. On

August 1, 2006, Dr. Jonathan C. Gibralter became president of Frostburg State

University.

Frostburg State University uses a computerized system, named PAWS, for its

student management, faculty management, and staff management needs. The PAWS

management system is built on PeopleSoft software and has been in operation for

approximately three years. It was assumed that the user already knew how to navigate to

the “Learning Management” page within PAWS. The “Learning Management” page is

similar to a homepage and is used by advisors to advise their students. Many users of this

interface have found themselves losing a lot of time looking for information in the wrong

place due to the lack of descriptive links. Therefore, they lose time as they drill down

through the pages looking for information that will not be there. Many users have

expressed their frustration when using the PAWS interface. Due to the level of

frustration expressed by many users of the system, a prototype was designed. The

prototype was designed using the collected comments about the original interface. This

paper will discuss usability testing and design of one task requested of the participant in

the usability study. Specifically, the user was asked to find the time(s) and section

number(s) of the offerings for Principles of Finance for the fall 2007.

Old PAWS Usability Test 3

Business Case

The Chief Information Officer (CIO) of Frostburg State University was consulted

before the start of the original site visits. The CIO was informed that the study would not

include any sensitive areas of the system like student information or financial

information. The CIO stated that because Frostburg State University was a public

facility, and that PeopleSoft was not a closed system of the university’s, there would not

be a reason to be concerned with the study. The CIO stated that the university had no

objections to the study and that they would be interested in seeing the results of the

PAWS interface study.

The Site Visit

The functions and management capability of PAWS is vast. It ranges from

managing the financial needs of the facility to enrolling in classes for the students. Terms

such as “inefficient, poor user interface, hard to use, confusing, and inappropriate use of

links” were often used when describing the PAWS interface. Thus, it was determined

that the focus of the site visit was to analyze the advising tools provided by the system for

the advising of students. Due to the complexity of the PAWS system the participants

were asked to perform one task with the PAWS interface. Specifically, the participants

were asked to find the times and section number(s) for the Principles of Finance for the

fall semester 2007. Refer to Appendix C for images of the PAWS interface.

The majority of the individuals that use PAWS considered themselves veteran

users. They had a good understanding of how PAWS worked; further, they knew about

the majority of the quirks with the interface. It was assumed that the users of PAWS

knew how to complete the tasks asked of them.

Old PAWS Usability Test 4

Site Visit Objectives

The objectives of the site visit were:

• To collect and understand any frustrations experienced by the participant.

• Observe problems participants encountered when performing the

requested task in the PAWS system.

• Understand the perceived usefulness of the retrieved content.

• Understand the mental model of the participant as it related to retrieving

the requested content.

Participants

Five individuals were solicited from the Department of Computer Science to

participate in this short study. The PAWS system has been in use by the university for

several years; thus, it was expected that the majority of the participants had a good

understanding of the system and its interface. It was also expected that all advisors on

campus had a basic understanding of how to use a web browser, how to navigate the

PAWS home page, and how to log into the system. All of the participants in the site

visits were selected from the Department of Computer Science. This made the site visits

convenient for both the observer and the participants. In order for the participants to

complete the task comfortably, all site visits were conducted in a location of their

choosing. All participants chose their office as the location for the site visit. All of the

participants were curious about what they were going to do. Upon finding out, all

participants made similar comments about how unpleasant the PAWS interface was to

use. They were excited that someone was finally doing a study on the interface. The site

visits were conducted between March 1 and March 6, 2007 with the participants being

Old PAWS Usability Test 5

contacted the day before or the day of the site visit. It was explained to the participants

that the study would require between 10 and 15 minutes of their time. A total of six

participants were interviewed. However, one participant opted out of the audio

recording; thus, only five of the participants’ data were analyzed. It was determined that

the observer could not collect all of the necessary observations without the audio from

that site visit.

Data Collection

A script was developed to assist in the collection of data during the site visit. The

script contained a brief description of the study along with appropriate questions to be

answered by the participant. In addition, the user was asked to describe what they were

doing as they were doing it, also know as the think aloud protocol. Below is the script

that was used during the interviewing process.

Name:_____________________ Location:_____________ Date:______________________ Time:_________________ Temp: Cool/Warm/Hot Lighting: On/Off Windowed/Non-Windowed Script: I would like to ask for your participation in a short study of the PAWS interface. The study will take between 10 and 15 minutes. Would you mind giving me 10-15 minutes of your time for this study? If the user agrees: I will be asking you a few questions first, and then I will be asking you to log into PAWS and perform a simple task. While you are performing the task, I will be observing your actions. I would also like to record our conversation, is this ok with you? If YES Turn on recorder Otherwise, continue without turning on recorder Otherwise: Thank you, have a great day! Questionnaire: What do you consider your level of experience with a computer? (Novice, Intermediate, Expert) (Questioning will stop if the user indicates that they can't operate a computer.) What do you consider your level of experience with a web browser? (Novice, Intermediate, Expert) What do you consider your level of experience with the FSU intranet? (Novice, Intermediate, Expert) What do you consider your level of experience with the PAWS interface? (Novice, Intermediate, Expert)

Old PAWS Usability Test 6

Do you have PAWS bookmarked in your web browser of choice? (Yes/No) If answer == NO Why don’t you have it bookmarked? Otherwise, continue TASK: We are going on to the task portion of the study. Could you please log on to your computer, open a browser, and log into the PAWS website? WAIT for user to complete requested tasks. Now I would like for you to perform the following simple task. While you are performing the task, I will be watching and listening. Should you have any comments about the interface as you are performing the task, please feel free to comment. I will not be able to assist you in performing the task. Are you ready? Task: Find the times and section number(s) for Principles of Finance for the fall 2007. Total Time: (Was filled in at a later time.) Please explain to me your thought process as you went through the requested task. Recode Observations What did you find frustrating about the tasks that I have asked you to perform? Be sure to record all of the participants’ response. Recode Observations If you could improve this task, how would you improve it? Recode Observations What are your general thoughts about the PAWS interface? Recode Observations Any other comments made by the user while performing the requested task? Recode Observations

For an example of a completed script from the site visits, refer to Appendix A. In

addition to the above script, an audio recoding device was used to capture as much

information as possible during the site visit. After the interview, the audio recording was

reviewed to fill in any gaps in data collection.

Site Visit Complications

During the site visit, three notable issues arose. One individual did not mention

that they had a meeting to attend shortly after the site visit was scheduled. That

individual may have let thoughts about being on time to their meeting interfere with the

site study. Another individual became very frustrated when they were unable to complete

the task and the observer had to help the individual get “unstuck.” Finally, the original

Old PAWS Usability Test 7

script stated that the administrator would not help in the completion of the task. After the

first two site visits were completed, it was realized that no assistance could lead to the

participant failing to complete the task. To capture the thoughts of the participant at these

frustrating moments, an adjustment was made in the script offering help to the participant

only after the participant became frustrated with the task and showed signs of giving up.

Analysis of Site Visit

Three methods were used to analyze the data collected from the site visits. The

analysis of the data was based on the methodologies discussed by Hackos and Redish

(1998). A vignette scenario was used to summarize the enormous amount of data

collected from the think aloud exercise performed by participants. A data matrix was

used to organize the data collected from the questionnaire portion on the site visit.

Finally a data flow diagram was used to illustrate the work flow of the task.

Vignette Scenario

Based on the written data and the audio recordings, vignette scenarios were

quickly created after each site visit. The following is a summary of the vignette scenarios

created after the interviews were conducted.

The participants were asked to use a think aloud protocol and say anything that

may “come to mind” as they progressed through the task. Several of the participants

became frustrated with many aspects of the PAWS interface (See Appendix C). For

example, several participants questioned the term code for spring 2007. The participants

also became frustrated with finding the abbreviation for a department and for a course.

After a few moments, it was apparent that many participants were becoming frustrated.

To keep the users interested in the study, the observer showed many if the participants

Old PAWS Usability Test 8

where to locate the term code for fall semester 2007, the department abbreviation, and the

course number. Several participants were unable to proceed to the next screen because

they were hitting enter instead of clicking the “search” button located on the screen. It

took the participants several minutes to complete the task. When asked what frustrated

them the most, the participants replied that there were many things. The icons used on

the screen do not offer any help as to what they are. Not being able to locate the “special

codes” such as the “term code” and the “course code” was especially frustrating. Finally,

the participants stated that it is unintuitive to click a “search” button after filling in a

field. It should be noted that sometimes the search button acts like a “submit” button.

Next, the participants were asked for suggestions to improve the task. The participants

suggested that there should be more descriptive buttons on the various pages. The

participants also stated that the icons are not representative of what they do and do not

have a help tag associated with them. If they were corrected, the site would instantly be

more intuitive. The participants also suggested that a drop down menu for selecting the

term code and course abbreviation may be helpful. Finally, the participants suggested

working on a way to make the interface more intuitive. For example, if you can not hit

enter, at least place the “submit” button closer to the task.

Data Matrix

A data matrix was created to help organize the data collected from the participant

into information (Hackos and Redish, 1998). The data was collected from the participant

during the questionnaire portion of the site visit. Table 1 shows the data collected from

the short questionnaire asked at the beginning of the observation. However, the data has

only been used to evaluate the speed at which the participants performed the requested

Old PAWS Usability Test 9

task. Perhaps the task time could be compared to the same data collected after the

interface has been revised. A further use for the data was realized after the usability test

was completed.

Table 1

User Data Matrix Basic Observations P1 P2 P3 P4 P5

Date 3/1 3/1 3/3 3/5 3/6 Location Office Office Home Office Office

Time 1.30p 11.20a 10.20p 4.00p 3.45p Windows Yes Yes Yes Yes Yes

Lights On Off On On On Questions

Experience Computers Intermediate Expert Expert Intermediate Expert Experience Browser Intermediate Expert Expert Intermediate Expert

Experience FSU Intranet Intermediate Intermediate Intermediate Intermediate Expert Experience PAWS Intermediate Intermediate Intermediate Intermediate Intermediate

Bookmarked PAWS Yes No Yes Yes Yes Calculated

Observations Task Time(Seconds) 285 221 121 195 380

Help? Yes Yes Yes Yes Yes

Old PAWS Usability Test 10

Data Flow Diagram

The last tool used to analyze the data collected from the site visits was a data flow

diagram (DFD). The DFD was created to help understand the flow of the task and to

better identify common issues from all participants. For example, in the DFD shown in

figure 1, all participants have difficulty identifying the “term” decision and the “subject”

decision. The DFD shows the options that the participant attempted to try to find the

code. It clearly illustrates the additional time required if a code for “term” or “subject” is

not known. At each decision on the DFD, the interface offers a solution to the decision at

the cost of additional steps and time. All of the participants noted in their suggestions

that a drop down box listing “codes” would be beneficial at these decision points.

Figure 1

Old PAWS Usability Test 11

Site Visit Analysis

The PAWS interface, with regard to locating the Principles of Finance for the fall

2007 semester, is unintuitive, bulky, and inefficient. All of the participants noted that

they strongly disliked the interface. The fastest participant was able to complete the task

in 121 seconds; and the slowest participant took 380 seconds. The site visit showed the

average time to completion of the requested task was 240.4 seconds. As observed in the

data matrix, all of the participants consider themselves to be, at minimum, an

intermediate user of the PAWS interface. Because the PAWS interface was unintuitive,

the majority of the suggestions revolved around making the interface more intuitive.

Many suggestions were offered for creating more descriptive buttons and using drop

down boxes at the major decision points. For clarification, the participants suggested

allowing the user to use the enter key instead of clicking a button would make the site

more intuitive.

Cognitive Factors

Donald Norman (2002) discusses seven principles that a designer should take into

consideration when designing a task. Below is a brief analysis of the task completed in

the study of the PAWS system as it relates to Norman’s seven principles.

The first principle suggests that the design of the task “use both knowledge in the

world and knowledge in the head” (Norman, 2002, p. 188). The task completed by the

participants required a basic understanding of how to use a computer, how to use a web

browser, how to use the Frostburg State University intranet, and how to use the PAWS

interface. Because the PAWS interface requires a web browser to complete this task, this

principle is inherent in the task as it relates to the participant. The task would not be used

Old PAWS Usability Test 12

by an individual that did not meet the minimum criteria of being able to use a web

browser.

The second principle states that the structure of the task should be as simple as

possible (Norman, 2002). As presented in the DFD in figure 1, the designer of this task

did not keep this task simple. The designer cloaked a simple task with excessive

decisions to be made by the participant, yet did not offer assistance for completing the

task. This forced the participant to perform extra searches in order to continue the

process, or even to look for the necessary information in a catalog. The modified version

of this task will offer a mechanism to help the participant make their decisions without

the need for any extra searches.

Norman’s third principle suggests that the designer should “make things visible”

(2002, p. 188). The original design of this task limits the participants’ ability to make

informed decisions due to lack of informative help menus, lack of directions, and vague

button labels. The revised task will include informative help menus when the participant

hovers over buttons and links. It will also include more informative labels for buttons.

The fourth principle instructs the designer to “get the mappings right” (Norman,

2002, p. 188). For this task, the majority of the mappings for the PAWS interface are

correct. The links are the proper colors and the buttons are labeled with the action that

they perform. However, several participants expressed dismay that they must click on a

button to perform an action. This suggests that there is a slight mapping issue with the

site. If the participant chooses to hit the enter key instead of clicking a button, the data

entered into the fields on the page is erased without explanation. Due to programming

constraints, the revised task may still require the participant to click a button. However,

Old PAWS Usability Test 13

if the participant hits enter instead, it will retain the participant’s information instead of

discarding the information entered into the fields.

The PAWS interface “exploits the power of constraints” as suggested by Norman

(2002, p. 189). The participant always knows where they are in the PAWS interface.

Pages are clearly labeled, and the participant is not redirected outside of the web site

upon clicking on a button or link. The revised web site will leave the current constraints

in place.

The principle most violated on the site is proper error detection and correction

(Norman, 2002, p. 189). As previously mentioned, should the participant make the

mistake of using the enter key instead of clicking a button, the task does not advance and

the participant must re-enter their information. An investigation will be performed to see

if the site will tolerate the participant using the enter key in addition to the clicking of the

buttons to progress to the next screen. Errors could be reduced with clear labeling of

buttons, and explanation of the functionality of search features in the site. The error

messages presented to the participant after a mistake are cryptic and offer little help as to

how to correct the error. The revised site will tolerate mistakes by retaining data entered

into fields. All error messages will offer a clear solution to the problem. Buttons will

have clear labels to indicate their functionality.

Norman’s (2002) final suggestion is to make sure that the device, in this case a

web site, is using documented standards. With the exception of the participant not being

able to use the enter key, the site uses web development standards. However, the site

may not comply with newer standards with regards to accessibility. Accessibility laws

and standards are outside the scope of this study.

Old PAWS Usability Test 14

The Prototype

It is clear that a designer did not create this task, nor did a designer have any part

in designing the PAWS interface. The redesign of the interface took into consideration

all of Norman’s principles for good design as discussed above. Armed with these

revisions, the PAWS prototype interface, see Appendix D, conforms to design criteria

that enable both experienced and inexperienced participants to navigate the task quickly,

efficiently, and intuitively. In addition to the images located in Appendix D, a

nonfunctional prototype can also be found at: http://www.mflinn.net/paws. Incorporating

the users request into the prototype have reduced the level of frustration experience by

the users when using the PAWS interface. Based on the participant’s input, the first thing

that was created was a revised data flow diagram to help determine proper work flow

(figure 2).

Old PAWS Usability Test 15

Figure 2 The Usability Test

Five individuals were solicited from the Department of Computer Science to

participate in the usability test. The PAWS system has been in use by the university for

several years; thus, it was expected that the majority of the participants had a good

understanding of the system and its interface. All of the participants in the usability test

were selected from the Department of Computer Science. This made the usability test

convenient for both the observer and the participants. In order for the participants to

complete the task comfortably, all site visits were conducted in a location of their

choosing. The same five individuals chosen for the site visit were asked to participate in

the usability test. This expedited the process of the usability test as all of the participants

Old PAWS Usability Test 16

were aware of the PAWS interface study; thus, the study did not need to be fully

explained to the participants again. The usability tests were conducted on April 13, 2007.

The participants were informed that the usability test would require between 10 and 15

minutes of their time.

Data Collection

A script was developed to assist in the collection of data during the site visit. The

script contained a brief description of the usability test along with appropriate questions

to be answered by the participant. In addition, the user was asked to describe what they

were doing as they were doing it, also known as the think aloud protocol. Below is the

script that was used during the interviewing process.

Name: Location: Date: Time: Temp: Cool/Warm/Hot Lighting: On/Off Windowed/Non-Windowed Script: I would like to thank you for participating in my short study of the PAWS interface. This is a follow us to that study. This is the usability portion of my study. If you recall from before, I asked you a few questions about the PAWS interface in general, and then I asked you to perform a short task using the interface. This time, I would like to ask for your feedback on the same task using a prototype of the PAWS interface. Although I will be showing you a nonfunctional prototype, it will still allow you to complete the task. The study will take between 10 and 15 minutes. Would you mind giving me 10-15 minutes of your time for this study? If the user agrees: I will be asking you a few questions first, and then I will be asking you to log in to the PAWS prototype and perform a simple task. While you are performing the task, I will be observing your actions. I would also like to record our conversation, is this ok with you? If YES Turn on recorder Otherwise, continue without turning on recorder Otherwise: Thank you, have a great day! Questionnaire: What do you consider your level of experience as it pertains to computer use? (Novice, Intermediate, Expert)

Old PAWS Usability Test 17

(Questioning will stop if the user indicates that they can't operate a computer.) What do you consider your level of experience as it pertains to web browser use? (Novice, Intermediate, Expert) What do you consider your level of experience with the FSU intranet? (Novice, Intermediate, Expert) What do you consider your level of experience with the PAWS interface? (Novice, Intermediate, Expert) Do you have PAWS bookmarked in your web browser of choice? (Yes/No) If answer == NO Why don’t you have it bookmarked? Otherwise, continue TASK: We are to the task portion of the study. I am going to ask you to perform a task in the PAWS prototype. Could you please log on to your computer, open a browser, and go to http://www.mflinn.net/paws. WAIT for user to complete requested tasks. Now I would like for you to perform the following simple task. While you are performing the task, I will be watching and listening. Should you have any comments about the interface as you are performing the task, please feel free to comment. Are you ready? Task: Find the times and section number(s) for Principles of Finance for the fall 2007. Total Time: Please explain to me your thought process as you went through the requested task. Record Observations What did you find frustrating about the tasks that I asked you to perform? Be sure to record all of the participants’ response. Record Observations If you could improve this task, how would you improve it? Record Observations What are your thoughts about overhauling the PAWS interface to and include aliases as this prototype did? Record Observations Other comments made by the user while performing the requested task: Record Observations

For an example of a script that was completed during the usability test, refer to

Appendix B. In addition to the above script, an audio recoding device was used to

capture as much information as possible during the site visit. After the interview, the

audio recording was reviewed to fill in any gaps in data collection.

Old PAWS Usability Test 18

Site Visit Complications

During the usability test, one notable issue arose. One participant suggested that

because the observer’s office was closer, that the usability test be conducted there using

the observer’s computer. The participant was unaware that mouse response time and

quickness of the mouse can vary depending on the user preferences. The participant was

capable of completing the usability test; however, there was a heightened level of

frustration due to the mouse “flying around the screen” as the user stated.

Analysis of the Usability Test

Two methods were used to analyze the data collected from the site visits. The

analysis of the data was based on the methodologies discussed by Hackos and Redish

(1998). A vignette scenario was used to summarize the data collected from the think

aloud exercise performed by participants. A data matrix was used to organize the data

collected from the questionnaire portion of the site visit.

Vignette Scenario

Based on the written data and the audio recordings, vignette scenarios were

quickly created after each usability test. The following is a summary of the vignette

scenarios created after the interviews were conducted.

The participants were asked to use a think aloud protocol and say anything that

may “come to mind” as they progressed through the task. All of the participants were

extremely happy with the prototype PAWS interface (See Appendix D). The participants

consistently commented on how intuitive the prototype was, stating that there was not

much to “think” about as they completed the requested task. Further, the users

commented on how much faster they could locate information in the prototype interface.

Old PAWS Usability Test 19

Finally, the users commented on the removal of the cryptic codes for semester,

department, and course. All of the participants stated that if the entire PAWS interface

were designed in a similar fashion to the prototype, that they would be inclined to use the

PAWS interface more often.

Usability Test Data Matrix

A data matrix was created to help organize the data into information that was

collected from the users during the questionnaire portion of the site visit (Hackos and

Redish, 1998). Table 2 shows the data collected from the short questionnaire asked at the

beginning of the observation. The data has been used to evaluate the speed at which the

participants performed the requested task. The data collected from the usability test will

be used later in this paper for comparison between the site visit and the prototype

usability test.

Table 2 Prototype Usability User Data Matrix

Basic Observations P1 P2 P3 P4 P5 Date 4/13 4/13 4/13 4/13 4/13

Location My Office Office Home Office Office Time 2.00p 1.30p 10.15p 4.15p 11.15a

Windows Yes Yes Yes Yes Yes Lights On Off On On On

Questions Experience Computers Intermediate Expert Expert Intermediate Expert

Experience Browser Intermediate Expert Expert Intermediate Expert Experience FSU Intranet Novice Intermediate Intermediate Intermediate Intermediate

Experience PAWS Intermediate Intermediate Intermediate Intermediate Intermediate Bookmarked PAWS Yes No Yes Yes Yes

Calculated Observations Task Time(Seconds) 26 21 15 17 22

Help? No No No No No

Old PAWS Usability Test 20

Usability Test Analysis

The prototype PAWS interface, with regard to locating the Principles of Finance

for the fall 2007 semester, is intuitive, easy to use, and efficient. The fastest participant

was able to complete the task in 15 seconds; and the slowest participant took 26 seconds.

The participant that took 26 seconds had difficulty controlling the mouse due to a high

sensitivity setting. With the exception of one participant, all of the participants consider

themselves to be, at minimum, an intermediate user of the PAWS interface. It should be

noted that two of the participants, P1 and P5, downgraded their level of experience with

the FSU intranet. The reason for this is unknown as it was not noticed until the side by

side analysis was performed on the site visit and the usability test (table 3).

Table 3 Side by Side Matrix Comparison

Observations P1(O) P1(P) P2(O) P2(P) P3(O) P3(P) P4(O) P4(P) P5(O) P5(P)

Date 3/1 4/13 3/1 4/13 3/3 4/13 3/5 4/13 3/6 4/13

Location Office My Office Office Office Home Home Office Office Office Office

Time 1.30p 2.00p 11.20a 1.30p 10.20p 10.15p 4.00p 4.15p 3.45p 11.15a

Windows Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes

Lights On On Off Off On On On On On On

Questions Experience Computers Intermediate Intermediate Expert Expert Expert Expert Intermediate Intermediate Expert Expert Experience

Browser Intermediate Intermediate Expert Expert Expert Expert Intermediate Intermediate Expert Expert Experience

FSU Intranet Intermediate Novice Intermediate Intermediate Intermediate Intermediate Intermediate Intermediate Expert Intermediate Experience

PAWS Intermediate Intermediate Intermediate Intermediate Intermediate Intermediate Intermediate Intermediate Intermediate Intermediate Bookmarked

PAWS Yes Yes No No Yes Yes Yes Yes Yes Yes Calculated

Observations

Task Time 285 26 221 21 121 15 195 17 380 22

Help? Yes No Yes No Yes No Yes No Yes No

The average time for completion of the requested task using the prototype was

20.2 seconds. Because the prototype PAWS interface was intuitive and was designed

using the participants site visit comment, there was little discussion about how to improve

Old PAWS Usability Test 21

the prototype further. Only one user made a suggestion for further improvement on the

prototype. The participant suggested that the home button should be placed in a different

spot then its current location.

Recommendations

Based on the results of the prototype PAWS interface usability test, it is

recommended that the entire PAWS interface be redesigned to be similar to the prototype

discussed in this report and found at http://www.mflinn.net/paws. This study has shown

that the current PAWS interface is unintuitive, time consuming and uses cryptic codes.

All of the participants stated that they dislike using the system due to these reasons,

among many others. The prototype PAWS interface had a positive response from the

participants. All of the users were accepting of the prototype and rejoiced in the

modifications from the original. The drop down menus that were used to eliminate the

cryptic codes was especially well received. Finally, on average, the users completed the

requested task 220.2 seconds quicker using the prototype than the original PAWS

interface.

It is my recommendation that the PAWS interface be redesigned using the

prototype as a reference. This study has proven that if the PAWS interface were

redesigned, users would be more inclined to use the system. If designed properly, the

users of the PAWS system could find the information they are looking for more quickly

and without the frustration levels observed during the site visit.

Old PAWS Usability Test 22

Appendix A

Participant Collection Sheets- Site Visit Sample Name: P1 Location: His Office Date: March 12, 2007 Time: 1.30 pm Temp: Cool/Warm/Hot Lighting: On/Off Windowed/Non-Windowed Script: I would like to ask for your participation in a short study of the PAWS interface. The study will take between 10 and 15 minutes. Would you mind giving me 10-15 minutes of your time for this study? If the user agrees: I will be asking you a few questions first, and then I will be asking you to log into PAWS and perform a simple task. While you are performing the task, I will be observing your actions. I would also like to record our conversation, is this ok with you? If YES Turn on recorder Otherwise, continue without turning on recorder Otherwise: Thank you, have a great day! Questionnaire: What do you consider your level of experience with a computer? (Novice, Intermediate, Expert) (Questioning will stop if the user indicates that they can't operate a computer.) What do you consider your level of experience with a web browser? (Novice, Intermediate, Expert) What do you consider your level of experience with the FSU intranet? (Novice, Intermediate, Expert) What do you consider your level of experience with the PAWS interface? (Novice, Intermediate, Expert) Do you have PAWS bookmarked in your web browser of choice? (Yes/No) If answer == NO Why don’t you have it bookmarked? Otherwise, continue

Old PAWS Usability Test 23

TASK: We are to the task portion of the study. I am going to ask you to perform a task in PAWS. Could you please log on to your computer, open a browser, and log into the PAWS website. WAIT for user to complete requested tasks. Now I would like for you to perform the following simple task. While you are performing the task, I will be watching and listening. Should you have any comments about the interface as you are performing the task, please feel free to comment. I will not be able to assist you in performing the task. Are you ready? When ready, mark time start:3.05 Task: Find the times and section number(s) for Principles of Finance for the fall 2007. When finished, mark time completed:7.42 Total Time: 5.40 I needed to prompt the user a little further to understand the question. I gave an example that an advisee enters the room and he/she needs to take their upper level business course, and they would like to take principals of finance. I again had to reprompt the question.... Please explain to me your thought process as you went through the requested task. Looked up information in the catalog Use the enter key instead of the “submit” button” Use of symbols instead of words... he is asking why the spy glass? I had to help find the semester number and the course abbreviation. What did you find frustrating about the tasks that I asked you to perform? Be sure to record all of the participants’ response. Finding the semester number is a frustrating, and well as finding the course number and department abbreviation. – Why can’t you just type in “fall 2007” Hit enter instead of the submit button. This would cause the forms to reset. The form to find the course was very confusing and perhaps too much information on the screen for the user. If you could improve this task, how would you improve it?

Old PAWS Usability Test 24

The Look-up button is very confusing... what are you looking up. This should be more straight forward and explain what the user is looking up. The searches should be more intuitive. View schedule of classes is fine and self explanatory. Perhaps put a drop down in, tell people what they are looking up... what is a basic look- up. After some time, it will list the things that he needs to know. “List of term” Change the button placement for submit and search. There should be a way to hit enter. The information the is on the page should be more descriptive. What are your general thoughts about the PAWS interface? Other comments made by the user while performing the requested task: College catalogue was used to locate the information before looked up on the PAWS system

Old PAWS Usability Test 25

Appendix B Participant Collection Sheets - Usability Test Sample Name: P1 Location: My Office Date: 4/13/2007 Time: 2.00p Temp: Cool/Warm/Hot Lighting: On/Off Windowed/Non-Windowed Script: I would like to thank you for participating in my short study of the PAWS interface. This is a follow us to that study. This is the usability portion of my study. If you recall from before, I asked you a few questions about the PAWS interface in general, and then I asked you to perform a short task using the interface. This time, I would like to ask for your feedback on the same task using a prototype of the PAWS interface. Although I will be showing you a nonfunctional prototype, it will still allow you to complete the task. The study will take between 10 and 15 minutes. Would you mind giving me 10-15 minutes of your time for this study? If the user agrees: I will be asking you a few questions first, and then I will be asking you to log in to the PAWS prototype and perform a simple task. While you are performing the task, I will be observing your actions. I would also like to record our conversation, is this ok with you? If YES Turn on recorder Otherwise, continue without turning on recorder Otherwise: Thank you, have a great day! Questionnaire: What do you consider your level of experience as it pertains to computer use? (Novice, Intermediate, Expert) (Questioning will stop if the user indicates that they can't operate a computer.) What do you consider your level of experience as it pertains to web browser use? (Novice, Intermediate, Expert) What do you consider your level of experience with the FSU intranet? (Novice, Intermediate, Expert)

Old PAWS Usability Test 26

What do you consider your level of experience with the PAWS interface? (Novice, Intermediate, Expert) Do you have PAWS bookmarked in your web browser of choice? (Yes/No) If answer == NO Why don’t you have it bookmarked? Otherwise, continue TASK: We are to the task portion of the study. I am going to ask you to perform a task in the PAWS prototype. Could you please log on to your computer, open a browser, and go to http://www.mflinn.net/paws. WAIT for user to complete requested tasks. Now I would like for you to perform the following simple task. While you are performing the task, I will be watching and listening. Should you have any comments about the interface as you are performing the task, please feel free to comment. Are you ready? Task: Find the times and section number(s) for Principles of Finance for the fall 2007. Total Time: 26 Seconds Please explain to me your thought process as you went through the requested task. The user commented on how fast the mouse was moving. The user participated in the usability test in the administrator’s office, on the administrator’s computer. There user stated that there was not any “real” thought to complete the task and was very happy. What did you find frustrating about the tasks that I asked you to perform? Be sure to record all of the participants’ response. The user again commented about the mouse, and that it moved too fast. The use did not find anything frustrating about the task the they were asked to perform If you could improve this task, how would you improve it? The user took a few moments to review the prototype after they had completed the task. They could not think of any improvements at this time, and stated that they would notify me if they thought of anything. What are your thoughts about overhauling the PAWS interface to and include aliases as this prototype did?

Old PAWS Usability Test 27

The user expressed great delight at the thought of the PAWS interface being overhauled to include the improvements in the prototype. The user stated that they would use the interface more often if the overall interface were more user-friendly and would use aliases, as the prototype does, instead of cryptic symbols, semester codes, and department codes. Other comments made by the user while performing the requested task: Although the user was not familiar with my computer, and the mouse was quite fast for their taste, they were still capable of completing the task in a quick and easy manner. The user suggested that the prototype be shown to the “powers that be” and encourage them to retools PAWS and take the prototype revisions into consideration.

Old PAWS Usability Test 28

Appendix C PAWS Interface

Old PAWS Usability Test 29

Old PAWS Usability Test 30

Old PAWS Usability Test 31

Old PAWS Usability Test 32

Appendix D Prototype PAWS Interface

Old PAWS Usability Test 33

Old PAWS Usability Test 34

Old PAWS Usability Test 35

Old PAWS Usability Test 36

Old PAWS Usability Test 37

References

Hackos, J. T., & Redish, J. (1998). User and task analysis for interface design. New York: Wiley.

Norman, D. A. (2002). The design of everyday things (1st Basic paperback. ed.). New

York: Basic Books.