paper

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

Performing Usability Evaluation on Multi-Platform

Based Application for Efficiency, Effectiveness and

Satisfaction Enhancement

https://doi.org/10.3991/ijim.v15i10.20429

Nik Azlina Nik Ahmad (), Nur Iman Mardhiah Hamid Universiti Kuala Lumpur, Kuala Lumpur, Malaysia

[email protected]

Anitawati Mohd Lokman Universiti Teknologi MARA, Shah Alam, Malaysia

Abstract—Usability evaluation is a crucial activity to contribute to a

higher standard of user experience. Without proper usability evaluation,

the risk of delivering a dissatisfying product is likely to increase. This

study conducted a usability evaluation on multi-platform applications

according to usability attributes defined in ISO 9241-11 standard: effi-

ciency, effectiveness, and satisfaction. The evaluation was conducted

using the think-aloud protocol on both platforms, web and mobile ap-

plications, and fifty participants. The results discovered the suitability

of think-aloud protocol on multi-platform usability evaluation and re-

vealed some usability issues. Therefore, to further enhance the users'

satisfaction, efficiency, and effectiveness, a set of improvement rec-

ommendations were proposed. Insights from this study provide a stand-

ard basis on the usability evaluation procedure, which can guide the us-

ability evaluation of multi-platform applications.

Keywords—ISO9241, multi-platform, usability attributes, mobile usability, usability evaluation

1 Introduction

Nowadays, where almost everything is digitized, technology plays a significant

fundamental role in amplifying the way humankind lives their life. Technology not

only advances to alleviate human beings, but it can also promote humanity to human

beings if correctly embraced. This study aims to perform a usability evaluation on a

multi-platform-based application that runs on the Web and Android platform to en-

hance its efficiency, effectiveness, and users' satisfaction. A multi-platform applica-

tion refers to an application that runs on two platforms or more, which enable each of

them to send and receive updates [1]. Multi-platform development focused on optimi-

zation by having the same code programmed among the versions of various platforms

iJIM ‒ Vol. 15, No. 10, 2021 103

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

[2]. Among the benefits of multi-platform are cost and time saving, while the mobile

type device is said to be more popular as they are more usable than others [3, 4]. Mul-

ti-platform applications could be between web and mobile, hybrid, or interpreted web

applications [5, 6, 7].

The application was developed to support the pet boarding system in terms of res-

ervation, package management, and reporting. The evaluation was conducted on both

platforms; the web and mobile application by examining seven main tasks of different

scenarios and collecting important attribute such as task completion (number of peo-

ple completed the task), time is taken to complete on each task, errors on performing

the task and satisfaction ratings given by participants. A post-evaluation questionnaire

was distributed, consisting of required demographic information and application usa-

bility feedback based on those main attributes: effectiveness, efficiency, and satisfac-

tion level. The usability level can be identified for the application, and hidden prob-

lems obstructing its usability potential can be revealed. Previous research on a pet

application was conducted [8], where a mobile application was developed to assist

users in finding a temporary pet adopter. The application support location detection to

find the nearest pet's medical assistance using a map. The application was described

as easy to use and allows its users to leave ratings. A study [9] involved a mobile app

development for a shelter to manage pet lost and found and promote pet adoption on

stray animals. Another study that focuses on pet adoption, which is equipped with a

pet product business, was also conducted [10]. A study conducted by [11] involved

creating a system to carry out animal-related tasks more efficiently in a temporary

animal shelter. However, none of the pet-related studies focused on evaluating the

application's usability that was implemented on multi-platform. Therefore, the study

aims to identify the aspects of consumer demands on multi-platform-based application

in the context of pet boarding under different user categories to discover the usability

issues by validating the effectiveness, efficiency, and user satisfaction level towards

the application.

2 Literature Review

ISO 9241-11 standard described usability as an attribute of quality in use and as a

result of perceived effectiveness, efficiency, and satisfaction [12]. The two most fre-

quent usability characteristics are effectiveness (how reliably users can accomplish

their objectives) and effectiveness (resources that can be utilized efficiently to accom-

plish that purpose [13]. There is a strong correlation between effectiveness and effi-

ciency. The simpler the interface is, the more resources can be devoted to other tasks,

which leads to task performance improvement. The study has noted a significant rela-

tionship between those two with user satisfaction, the beneficial outcome of pleasant

interactions, including feelings of satisfaction. Assessing the usability involved meas-

uring user interaction performance with the system, which involves efficiency, effec-

tiveness, and error rate [14]. Since it is newly designed, the user satisfaction level is

vital in how the application portrays; its first impression can affect its decision to put

it into fair use. Hence, usability evaluation is an effective way to measure the applica-

104 http://www.i-jim.org

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

tion's performance and possibly search for improvement rooms. Furthermore, explor-

ing various similar studies on how the usability assessment can be done helps decide

to choose a suitable method. In this case, usability testing is the best approach as it is

a commonly used technique that had been proven to be effective from various usabil-

ity evaluation research studies as it encourages feedback from the end-user.

From various usability evaluation studies, the overall usability aspect differs for

each study, where each brings a refreshing perspective of its' own. Usability is a par-

ticular attribute that influences the software system's consistency and should satisfy

user requirements [15]. It complements user experience as usability focuses on the

functional part while user experience is emphasized on the emotions stemmed from

the aspects of the product, as explained in [16]. It considers user satisfaction as quality

indicators and aims to improve user experience. In a way, it relates to work conducted

[17], which utilized three attributes that are considered vital as a measurement of how

a user can use software to complete a set of operations successfully. These attributes

are efficiency, effectiveness, and user satisfaction. Besides, a mobile application's

usability challenges include limited connectivity in certain areas, small screen size,

small storage capability, and the unconventional input mechanism. Furthermore, [18]

stated that usability should be approached from multiple points of view to become

sensitized to elements that possibly affect the system usage. On the other hand, usabil-

ity is characterized as how successful user concerns are addressed. [19]. Hence, in-

stead of adopting the standardized usability model, many studies include attributes

considered important in developing any application. For example, in the study con-

ducted [20], usability is vital to increase application accuracy and reduce reaction

time for several user activities. Overall, the various definition of usability stated above

actually describes the advantages it brings indirectly. To summarize, there are plenty

of benefits of implementing usability evaluation, such as reducing training effort,

improving the product quality, and users' satisfaction.

In terms of usability attributes to pay attention to, various usability evaluation stud-

ies implemented standardized usability models, as presented in Table 1. Besides that,

usability evaluation studies only implement attributes vital in the application-specific

context without following any specific usability models. There are also usability eval-

uation studies that combined selected usability models and attributes vital to the dedi-

cated application.

iJIM ‒ Vol. 15, No. 10, 2021 105

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

Table 1. Variety of usability attributes

No. Usability

Attributes Description Source

1 Effectiveness, Efficiency and

Satisfaction.

The international organization of standardization came up with the ISO

9241-11 model, which consists of 3 attributes: a performance measure

of completing tasks on time, efficiency in completing the task success- fully, and satisfaction as users' acceptability.

[21],

[22],

[18], [17].

2

Learnability,

Efficiency, Memorability,

Errors and

Satisfaction.

Jakob Nielsen model consists of five attributes, which are learnability as

in easy to learn, efficiency as in easy to finish a task or navigate, memo-

rability as in easy to remember or reestablish, errors as in the low rate of error detection and satisfaction as in it is pleasant to use by the user.

[16],

[22],

[15], [23].

3

Functionality

Conformance, Efficiency, Ease

of learning,

Memorability, Correctness and

Esthetics.

Factors in the article journal include compliance with features specified

as ISO software operability, efficiency or latency, ease of learning as in understandable, memorability as easy to remember, correctness in lieu

to dealing with system failure, and esthetics as attractive.

[19]

4 Errors, Under- standability,

Accessibility.

In addition to effectiveness, efficiency, and satisfaction, these additional factors were used to assess mobile applications and the visually im-

paired.

[24]

5

Adjustability,

Inability, Relia-

bility and Satis- faction.

Factors include adjustability that refers to the degree of the user accept-

ability to the platform used to learn, inability as in a mobile learning

platform, and the concept pleased the users. Reliability in a mobile

learning platform means that the platform should not perform unex-

pected ways and satisfaction as in learner's satisfaction and the learner's

assurance to perform tasks with any external activities via their mobile device.

[21]

6

Effectiveness,

Efficiency and Learnability.

The researchers decided to exclude satisfaction as an attribute and absorbed learnability to ensure a more parsimonious framework.

Attributes learnability refers to how much learning a system is conven-

ient for users.

[25]

7

Effectiveness,

Learnability,

Efficiency, Memorability,

Errors, Cognitive

Load, Satisfac- tion, and Timeli-

ness.

It is a combination of the PACMAD usability model developed by

Harrison and a single attribute that was deemed vital in mobile learning,

in which timeliness refers to whether students can receive teachers' messages quickly. PACMAN attributes such as learnability, efficiency,

memorability, error, and satisfaction are absorbed from Jakob Nielsen's

model, where all of them serve the same purpose. The cognitive load attribute relates to the number of perceptual abilities needed by the

program when being utilized.

[20]

2.1 Usability assessment

From various researches on various usability evaluation, there is more than one

way to conduct the assessment. Usability evaluation can be conducted using various

methods such as hallway usability testing, a cheap method in which a group of ran-

dom people will be selected to do a set of tasks. Users will interact with the system

while thinking aloud. It is a commonly used technique that helped numerous usability

evaluation research [16, 26]. The expert review involves usability experts using a set

of guidelines to measure essential usability criteria, while an automated expert review

is a similar procedure of expert review but in an automated way [16]. Another as-

sessment method is inspection, where users act as an observer while experts do evalu-

106 http://www.i-jim.org

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

ation and testing. Inspection can be done by a heuristic evaluation, which utilizes the

usability principle to implement. It is cheap or conducted by a cognitive walkthrough

based on user interface evaluation by experts that consider end-user opinions [22, 23].

Cognitive walkthrough is similar to heuristic evaluation because experts do it, but it

does not necessarily require a whole system. A system prototype is just adequate.

However, a pluralistic walkthrough [27, 28, 29] is a method for a team to investigate

the system's paper prototype with the users' focus group to discuss system interface

issues [26]. [16] however, highlighted the non-face-to-face assessment by suggesting

remote usability testing. This method separates test conductor and user and is catego-

rized by the synchronous time such as remote application sharing software or asyn-

chronous time where data will be collected from logging user's activities. Further-

more, there are a different kind of data gathering techniques can be utilized to extract

the outcomes of the evaluation which can either be test monitoring, direct recording, a

think-aloud method which require the user to talk aloud while using the system or

performing the particular task as explained in [23]. On top of that, there are also other

methods of conducting usability testing, such as Metrics for Usability Standards in

Computing by (MUSiC), which was introduced in 1998 [30], Software Usability

Measurement Inventory (SUMI). In 1992, Diagnostic Recorder for Usability Meas-

urement (DRUM) by [31] was introduced.

Usability assessment has been conducted in various aspects such as counseling

[32], education [20, 21, 23, 24, 25, 33], shopping [16, 17], tourism [18], Web portal

[15, 19], and visual impairment study [24]. However, minimal pet-related studies

concentrated on evaluating the usability in the multi-platform context.

3 Methodology

The usability evaluation was conducted on both platforms, web and mobile plat-

forms, measuring the efficiency, effectiveness, and satisfaction outlined in ISO 9241-

11. The usability assessment framework for a multi-platform application used in this

study is presented in figure 1.

Fig. 1. Usability assessment framework for multi-platform application

iJIM ‒ Vol. 15, No. 10, 2021 107

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

3.1 Data collection

The methodology involved hiring the test participants, preparing a test plan, and fi-

nalizing the test scenario according to both platforms' set of tasks. Necessary technical

preparation was made, which include device and site set up. Data was collected using

pre-testing, post-testing questionnaire, observation technique, and think-aloud proto-

col with fifty participants. A pre-testing questionnaire was used to collect demograph-

ic data, whilst the post-testing questionnaire was equipped with 15 questions to identi-

fy the participants' satisfaction level. The questionnaire's findings were objectively

analyzed, interpreted as a mean rate of satisfaction calculation, while the outcome of

the think-aloud protocol was evaluated and categorized according to the right usabil-

ity category as recommended [12] in order to determine efficiency and effectiveness

and provide relevant recommendations.

3.2 Usability evaluation session

The session was carried out in a pet boarder's place with a total of 50 participants.

The participants were randomly selected from the existing pet boarder's customers

and non-customers with ages ranging from 18 to 64. An initial demographic back-

ground was surveyed before proceed with the evaluation task. Participants with the

following criteria were selected: a pet boarder who owns and offers boarding service,

regular visitors to the pet store, a customer to pet border, and those pets. A set of tasks

for the different scenario was identified for the evaluation. There were two types of

scenarios representing the pet border (web-based) and customer role (mobile-based).

A total of 8 task scenarios were prepared for the pet border, and 7 for the customer's

role. All participants will act in different roles and access the application on both

platforms with two appointed facilitators' assistance. All evaluations used the re-

searchers' laptops and mobile devices. Participants were allocated 4 minutes to ac-

complish each task scenario. The task scenarios for each role are portrayed in Table 2.

108 http://www.i-jim.org

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

Table 2. Evaluation task scenario on both platforms

Task No. Web-Based

Pet Boarder Task Scenario

Mobile-Based

Customer Task Scenario

1

The user clicks the signup button and will be redirected to the registration page. User input

username, name, email address, system role, password, confirm password, check the term, and

policy agreement checkbox. Finally, click the Register button.

2 User input username, password, and click sign in button.

3

User clicks the create announcement menu

section, input the message, and click the create button.

The user clicks the announcement menu

section and the view list of the announce- ment.

4 User clicks the create notification menu section,

input the message, and click create button.

Users click the notification menu section, and the list of all notifications posted will be

listed.

5

User clicks the create package menu section,

input the package name, slot available, price,

reward points, upload multiple pictures, and add one addon service.

User clicks the package menu section whereby all packages will be listed, choose

one package, click the book now button,

input pet name, select start date and end date, select available addon service check-

box button and click create button.

6

User click booking menu whereby all booking

requests made by all customers will be listed,

select one booking request and click the update

button, view booking status information, select

approved from the dropdown list, and click the

update button.

User view list of booking requests made by

clicking on the booking request menu

section, and the list of all booking requests

made by the user is listed.

7

The user clicks the report card menu whereby all booking requests made by each customer will be

listed, choose one booking request and click the

create report button, input notes, upload media files by clicking the add media icon or choose file,

select a picture and click the insert button.

The user clicks the report card menu section whereby all report card submitted is

displayed; the user can view the pet name

and report date, click to view the notes button, and click to view the media files

button.

8

User clicks the user menu whereby all users will

be listed, select one user, click the update button, scroll down to the user status information, the

approved user from the dropdown list, and click

the update button.

N/A

For collecting numerical data, a survey has been conducted as a second evaluation.

There were two sets of distributed questionnaires, consisting of the respondent back-

ground information and application review from the end-user. The usability question-

naires include the following: 1) I think this application is easy to use, 2) It is easy to

find the information needed, 3) I think this application has a pleasant interface, 4) The

font utilized by the application is pleasant, 5) This application can speed up task com-

pletion, 6) This application provides error messages that inform how to solve the

problems, 7) I find the steps to execute tasks in this application is simple, 8) This

application provides clear and descriptive information, 9) I feel that it is easy to navi-

gate through the pages, 10) This application has no broken link/menu/page, 11) The

iJIM ‒ Vol. 15, No. 10, 2021 109

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

application provides all the functions that I expected it to have, 12) This application

needs to be improved, 13) I have the intention to use this application shortly, 14)I will

recommend this application to others, and 15) Overall, I am satisfied with this appli-

cation.

4 Results and Discussion

The study was carried out by issuing questionnaires to the 50 participants after

each respondent had done the set of tasks required for the usability evaluation. The

purpose of choosing participants from different backgrounds is to have a broader

perspective of the study.

Fig. 2. Distribution of respondents by profession

As shown in Figure 2, respondents were categorized according to their occupation

status, working in the private sector, government servant, self-employed, retiree, and

student. The highest number was those who worked in the private sector reflected the

ability to own pets, requiring cost and maintenance.

Task Completion Time: 100% of participants completed task 1 until four on both

platforms, representing sign up, sign in, and create / view announcement. Only 41 out

of 50 users can complete task 5, create a package by admin on the web (82.0%) with

seven errors, while 90.0% can complete the view/apply package on mobile with four

errors. No issues were reported on task 6 on mobile for viewing customer's booking

request, but three errors have been identified on the web application for approve

booking request scenario with a 94% completion rate. Task 7, which focuses on up-

dating the report card, has encountered four errors and a 92% completion rate, while

the mobile application has a 100% completion rate with zero error encountered. The

final task, which is approved user registration, is only available on the web, and this

task has triggered five errors with an 86.0% completion rate. This is because the user's

Student 20%

Government 24%

Private 36%

Self- Employed,

14%

Retired 6%,

110 http://www.i-jim.org

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

name did not appear in the list even after several pages reloads. Task completion time

is a measure of the efficiency of the app. Table 3 displays the time taken by each user

per task compared to the allowed 3 minutes for the task completion—more time spent

on Web-based applications than mobile applications due to a more user-friendly look

at mobile. The time allocated for each task was 4 minutes. However, not all the tasks

can be completed within the allocated time: tasks 3, 5, 6, 7 for web-based applications

and task 5 for mobile applications. The lowest completion time was spent on task 5,

which was 4.7 minutes reflecting that the application lacks efficiency.

Table 3. Summary of task completion, errors, and time on task

Task No.

Web-Platform Mobile-Platform

Task

Completion Errors

Average Time

on Task Task Completion Errors

Average Time

on Task

1 50 0 2.1 min 50 0 1.6 min

2 50 0 2.4 min 50 0 2.3 min

3 50 0 3.2 min 50 0 2.9 min

4 50 0 3.0 min 50 0 2.5 min

5 41 7 4.7 min 45 4 2.7 min

6 47 3 3.5 min 50 0 3.4 min

7 46 4 4.6 min 50 0 3.1 min

8 43 5 2.4 min - - -

Success rate of task completion: Task completion or success rate is a measure of

an application's effectiveness in use. Table 3 shows that only tasks 1 – 5 achieved a

100% completion rate while the rest scored below 94%, which is equal to 47 partici-

pants and below. Therefore, this application is not sufficient enough. The age might

be the factor of this result as 3 participants were retirees with age above 55. Elderly

participants took a long time getting familiar with the interface and understanding the

steps even though they are smartphone users. Table 4 summarizes details on partici-

pants' error and feedback, classified according to ISO 9124-11 usability attributes;

efficiency, effectiveness, and satisfaction. A set of recommendations was proposed to

promote the application's enhancement, leading to higher efficiency and effectiveness

levels.

iJIM ‒ Vol. 15, No. 10, 2021 111

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

Table 4. Errors, Comments, and Recommendations for Improvements

Attributes Errors / Comments Recommendations

Efficiency

Task 5 (Web)

• “It takes me quite a while to figure out what to put for as addon.”

Task 6 (Web)

• "It is confusing because there are so many users to go through (navigating)."

Task 5 (Mobile)

• "I have to spend a long time as it is tricky for me to figure out how to input

the date with the proper format because

I kept getting an error message."

• "Menu section to make a booking is label as History. Hence I am confused."

Task 3 and 4 :

• Distinguished the interface design for announcement and notification to show each serves a different purpose.

• Create a text input for the text field whereby it gives an example of an ap-

propriate message for an announcement

or notification. Task 5 :

• Place a pop-up containing a few sugges- tions on what kind of addons services are

commonly offered in the pet industry.

This will help the user to make a faster decision.

• A visible pop calendar icon at the end of the text field can easily pick the dates.

• Create a text input for the text field whereby it alerts the user to click the icon nearby.

• Each package available on display should show written information such as

any addons available or reward points.

• Menu section History should be changed to Booking and contains a subsection

called My History. Task 6:

• Place a filter button whereby the user can utilize this function to filter out the cus- tomer's pending registration status.

• It is better to provide a help needed button whereby if it is clicked, it displays

a video on how to approve the customer.

Effectiveness

Task 5 (Web)

• “I have no idea what to put for addons for the package, so I did not put any-

thing extra."

Task 6 (Web)

• "It just shows the status, and there is just an update button instead of approv-

ing button." Task 5 (Mobile)

• "I cannot input the dates for booking."

• “I was unaware of the aid provided by the calendar date selection.”

Satisfaction

Task 3 and 4 (Web)

• "There is no problem, but I cannot differentiate between what to post an

announcement or notification."

Task 5 (Web)

• "I am moderately satisfied." Task 5 (Mobile)

• "The interface is too technical," "sim- ple."

• The information provided on the menu section and each package is unclear and

written poorly."

Satisfaction: Besides evaluating the applications using the prepared tasks list, par-

ticipants must complete a usability questionnaire using a five-point Likert scale, rang-

ing from 1 to 5, from strongly disagree to agree strongly. The questions were to eval-

uate the participants' satisfaction over the completed tasks performed earlier. Table 5

summarizes the findings.

112 http://www.i-jim.org

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

Table 5. Satisfaction feedback

General Strongly

Agree Agree Neutral Disagree

Strongly

Disagree

%

Agree

I think this application is easy to use. 17 21 11 1 0 76%

It is easy to find the information needed. 15 19 13 3 0 68%

I think this application has a pleasant interface. 6 15 17 10 2 42%

The font utilized by the application is pleasant

for my eyes. 8 15 21 6 0 46%

This application can speed up my task comple-

tion 13 24 11 2 0 74%

This application provides error messages that tell

me how to solve the problems. 13 20 15 2 0 66%

I find the steps to execute tasks in this application

is simple. 9 23 16 2 0 64%

This application provides clear and descriptive information.

1 20 25 2 2 42%

I feel that it’s easy to navigate through the pages. 1 19 22 6 2 40%

This application has no broken link/menu/page. 11 22 14 3 0 66%

The application provides all the functions that I expected it to have.

13 19 16 2 0 64%

This application needs to be improved. 13 12 23 2 0 50%

I have the intention to use this application shortly 5 17 25 3 0 44%

I will recommend this application to others. 7 15 22 6 0 44%

Overall, I am satisfied with this application. 12 23 14 1 0 70%

Fig. 3. Overall Mean Analysis

Overall, most of the participants were satisfied with the application, with an aver-

age mean of 3.68. The agreed notch counts the agree and strongly disagree score. 70%

of participants agreed that they were satisfied using the application on both platforms.

The highest score was on the ease of use, with 76% agreement, followed by the appli-

cation's ability to speed up participants' task with 74%, while the lowest score accord-

ing to participants was on the page navigability, 40%. Due to several crucial errors

during the evaluation, half of the respondents thought this application needs modifica-

tion. Suggestions for modification have been identified in Table 4 to increase the

users' satisfaction by further adopting the formal requirements technique [34]. The use

4.08 3.92 3.28 3.5

3.96 3.88 3.78 3.32 3.22 3.82 3.86 3.72 3.48 3.46 3.92

0

2

4

6

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15M e

a n

R a

ti n

g

Question Number

Mean Data Value

iJIM ‒ Vol. 15, No. 10, 2021 113

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

of the think-aloud protocol was proven to help obtain a comprehensive understanding

of usability issues [35] related to the user and multi-platform application interactions,

where 12 crucial errors from 8 assigned tasks were successfully discussed.

5 Conclusion

This study has explored the potential use of the think-aloud protocol in determining

the usability of the multi-platform framework to optimize its efficiency, effectiveness,

and satisfaction. This study's proposed approach offers new insights in the usability

evaluation context, particularly for mobile and multi-platform applications. The eval-

uation involved both platforms, web and mobile applications, and the users were re-

quired to evaluate the application in both platforms using a think-aloud protocol. This

study involved measuring the success rate of task completion, task completion time,

task errors, participants' feedback, and a satisfaction assessment through a question-

naire with the involvement of 50 participants. The results indicated that 100% task

completion rate was only applied to task 1 until task 4 for both platforms, and an

additional task 7 for mobile application. The remaining tasks discovered errors, and

comments from participants were recorded. Therefore, this application seems to lack

efficiency and effectiveness. However, satisfaction has a good score, with an average

mean of 3.68. To further increase the satisfaction rate, efficiency, and effectiveness,

the suggestion for improvements were made accordingly. It is believed that fixing the

errors as highlighted by participants will likely increase the satisfaction, efficiency,

and effectiveness of the application on both platforms. This study's limitation in-

volved a biased sample that focuses on a specific pet border, and user satisfaction

based on human needs behavior was not recorded. Due to a severe need for useful

tools [36], future studies are expected to produce comparable outcomes after specific

improvements were made on the user interface based on the results of this study [37,

38] and present the model requirements for the reliable modeling of actual user be-

havior to extract usage patterns.

6 References

[1] Nik Ahmad, N. A., and Syed Zamri, S. A. (2014). The cross-platform application devel- opment adapted the Spring framework to support front-end tendering services. 2014 Inter-

national Conference on Computer, Communications, and Control Technology (I4CT),

Langkawi, 2014, pp. 58-62, https://doi.org/10.1109/i4ct.2014.6914145

[2] Delia, L., Galdamez, N., Thomas, P., Corbalan, L., and Pesado, P. (2015). Multi-Platform Mobile Application Development Analysis, 2015 IEEE 9th International Conference on

Research Challenges in Information Science (RCIS), https://doi.org/10.1109/rcis.2015.712

8878

[3] Papadakis, S.; Vaiopoulou, J.; Kalogiannakis, M.; Stamovlasis, D. (2020). Developing and Exploring an Evaluation Tool for Educational Apps (ETEA.) Targeting Kindergarten

Children. Sustainability, 2020, 12, 4201. https://doi.org/10.3390/su12104201 [4] S. Papadakis, and M. Kalogiannakis, “A research synthesis of the real value of self-

proclaimed mobile educational applications for young children.,” Mobile learning applica-

114 http://www.i-jim.org

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

tions in early childhood education, 1-19, 2020. https://doi.org/10.4018/978-1-7998-1486- 3.ch001

[5] El-Kassas, W. S., Abdullah, B. A., Yousef, A. H., & Wahba, A. M. (2017). Taxonomy of Cross-Platform Mobile Applications Development Approaches. Ain Shams Engineering

Journal, 8(2), 163–190. https://doi.org/10.1016/j.asej.2015.08.004

[6] Martinez, M., & Lecomte, S. (2017). Towards the Quality Improvement of Cross-Platform Mobile Applications. 2017 IEEE/ACM 4th International Conference on Mobile Software

Engineering and Systems (MOBILESoft). https://doi.org/10.1109/mobilesoft.2017.30

[7] Kadhim KJ, Odhaib MF, Hadi YH, Ameen HM, Muhdi AA, et al. (2018) Developing a Multi Platforms Web Applications for Mobile Device Using HTML5. J Inform Tech Softw

Eng 8: 225. https://doi.org/10.4172/2165-7866.1000225

[8] Saswadkar, V., Paygude, V., Dudhe, P., Garad, P., and Gaikwad, R. (2018). Pet Care Sys- tem Based on Android Application. International Journal of Trend in Scientific Research

and Development (IJTSRD), pp. 1488-1491. https://doi.org/10.31142/ijtsrd14415 [9] Kolandaisamy, R., Subaramaniam, K., Kolandaisamy, I., and Siew, L. (2016). Stray Ani-

mal Mobile App. Conference Paper UCSI University, pp. 1-7.

[10] Santy, Karuna, R., and Budiman, A. (2018). E-dopt: A Mobile Application for Pet Adop- tion in Indonesia. TELKOMNIKA. pp. 2137-2143. https://doi.org/10.12928/telkomni

ka.v16i5.8074

[11] SENER, R., and TURK, T. (2019). Developing an Android-based Mobile Application for Temporary Animal Shelter Activities. Eurasian Journal of agricultural research. pp. 1-8.

[12] ISO 9241-11:2018 (en) Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts.

[13] C. Coursaris & W. Van Osch, (2016). A Cognitive-Affective Model of Perceived User Sat- isfaction (CAMPUS): The complementary effects and interdependence of usability and

aesthetics in IS design. Information and Management, 53 (2), 252–264. https://doi.org/10.

1016/j.im.2015.10.003

[14] Sulaiman, S., Mahmood, A., & Khan, M. (2015). Understanding factors influencing user experience of interactive systems: a literature review. Journal of Engineering and Applied

Sciences, 10 (23), 18175–18185.

[15] Benmoussa, K., Laaziri, M., Khoulji, S., Larbi Kerkeb, M., and El Yamani, A. (2019). Evaluating the Usability of a Moroccan University Research Management Web Platform.

Procedia Manufacturing. Pp. 1008-1016. https://doi.org/10.1016/j.promfg.2019.02.315 [16] Erik, B. (2014). Usability testing of e-shop software. Institute of Computer Science Infor-

matics Curriculum. Pp. 1-50.

[17] Hussain, A., E. O. C. Mojiogu, Jamaludin, N.H, Somia, and T. L. Moh (2017). A usability evaluation of Lazada mobile application. The 2nd International Conference on Applied

Science and Technology. Pp. 1-6. https://doi.org/10.1063/1.5005392 [18] Groth, A., and Haslwanter, D. (2016). Efficiency, effectiveness, and satisfaction of respon-

sive mobile tourism websites: a mobile usability study. Information Technology and Tour-

ism. Pp. 201-228. https://doi.org/10.1007/s40558-015-0041-0

[19] Ali-Shahid, M.M., and Sulaiman, S. (2015). A Case Study on Reliability and Usability Testing of a Web Portal. Malaysian Software Engineering Conference. Pp. 31-36.

https://doi.org/10.1109/mysec.2015.7475191

[20] Parsazadeh, N., Ali, R., Rezaei, M., and Tehrani, S. Z. (2018). The construction and vali- dation of a usability evaluation survey for mobile learning environments. Studies in Educa-

tional Evaluation. Pp. 97- 111. https://doi.org/10.1016/j.stueduc.2018.06.002

[21] Hujainah, F., Dahlan, H., and Al-Haimi, B. (2013). Usability Guidelines of Mobile Learn- ing Application. Journal of Information Systems Research and Innovation. Pp. 70-77.

[22] Madan, A., and Kumar Dubey, S. (2012). Usability Evaluation Methods: A Literature Re- view. International Journal of Engineering Science and Technology (IJEST). Pp. 590-599.

iJIM ‒ Vol. 15, No. 10, 2021 115

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

[23] Azhari, S.A., Redzuan, F., and Abdul Majid, R. (2015). Usability Evaluation of Academic Record System at the International Islamic College (IIC). IEEE Conference on e-Learning,

e-Management and e-Services. Pp. 1-6. https://doi.org/10.1109/ic3e.2015.7403493

[24] Hussain, A., and Omar, A.M. (2020). Usability Evaluation Model for Mobile Visually Im- paired Applications. International Journal of Interactive Mobile Technologies (iJIM). Vol.

14, No. 05:95. https://doi.org/10.3991/ijim.v14i05.13349

[25] Joo, S., Lin, S., and Lu, K. (2011). A Usability Evaluation Model for Academic Library Websites: Efficiency, Effectiveness and Learnability. Journal of Library and Information

Studies. Pp. 11-26.

[26] Zhao L, Loucopoulos P, Kavakli E, and Letsholo K (2019) User Studies on End-User Ser- vice Composition, ACM Transactions on the Web, Online Publication, Vol. 13, Issue 3,

pp. 1-46. https://doi.org/10.1145/3340294

[27] Riihiaho, S. (2001). Experiences with Usability Evaluation Methods. Helsinki University of Technology.

[28] Zhang, T.; Booth, R.; Jean-Louis; R., Chan, R.; Yeung, A.; Gratzer, D.; Strudwick, G. (2020). A Primer on Usability Assessment Approaches for Health-Related Applications of

Virtual Reality, JMIR Serious Games 2020; Vol. 8, Issue 4: e18153. https://doi.org/

10.2196/18153

[29] Devyatkov, V. V., and Tipsin, E. A. (2020). User Behavior Model Development in The Tile Logic for Automating Usability Evaluation. International Conferences Computer

Graphics, Visualization, Computer Vision and Image Processing 2020, 23 – 25 July 2020,

Russia. https://doi.org/10.33965/cgv2019_201906l032 [30] Macleod, M., Bowden, R., and Bevan, N. (1998). The MUSiC Performance Measurement

Method. National Physical Laboratory. Pp. 2-26.

[31] MacLeod, M., & Rengger, R. (1998). The Development of DRUM: A Software Tool for Video-assisted Usability Evaluation. Computer Science.

[32] Nik Ahmad, N.A., and Syed Dzulkarnain, S.N.S. (2020). Utilization of Gardner’s Multiple Intelligence Theory for School Counselling System with Usability Testing. International

Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Is-

sue-6, pp. 2253 – 2260. https://doi.org/10.35940/ijrte.e6058.038620

[33] Hussain, A., E. O. C. Mkpojigou, and Babalola, E.T. (2020). Using Mobile Educational Apps to Foster Work and Play in Learning: A Systematic Review. International Journal of

Interactive Mobile Technologies (iJIM). Vol. 14, No. 18. Pp. 178-194. https://doi.org/

10.3991/ijim.v14i18.16619

[34] Ahmad, N. A. N., and Kasirun, Z.M. (2011), ‘Elicitation strategies for Web application us- ing activity theory. Journal of Advances in Computer Research. Volume 2, Number 3 (5),

pp. 1-13 [online]. Available: https://www.sid.ir/en/Journal/ViewPaper.aspx?ID=260575

[35] Cho, H., Powell, D., Pichon, A., Kuhns, L. M., Garofalo, R., & Schnall, R. (2019). Eye- tracking Retrospective Think-aloud as a Novel Approach for a Usability Evaluation. Inter-

national Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2019.07.010

[36] Papadakis, S. (2020). Tools for evaluating educational apps for young children: a system- atic review of the literature. Interactive Technology and Smart Education, Vol. ahead-of-

print No. ahead-of-print. https://doi.org/10.1108/itse-08-2020-0127

[37] Drolia, M.; Sifaki, E.; Papadakis, S.; Kalogiannakis, M. (2020). An Overview of Mobile Learning for Refugee Students: Juxtaposing Refugee Needs with Mobile Applications’

Characteristics. Challenges,11(2), 31. https://doi.org/10.3390/challe11020031

[38] Papadakis, S., & Kalogiannakis, M. (2020). A Research Synthesis of the Real Value of Self-Proclaimed Mobile Educational Applications for Young Children. In Mobile Learning

Applications in Early Childhood Education (pp. 1-19). IGI Global. https://doi.org/10.4018/ 978-1-7998-1486-3.ch001

116 http://www.i-jim.org

Paper—Performing Usability Evaluation on Multi-Platform Based Application for Efficiency and…

7 Authors

Nik Azlina Nik Ahmad is a lecturer at the Department of Software Engineering in

Universiti Kuala Lumpur. She has experience working as a system analyst for a li-

brary system development. Her research interests lie in software testing, usability,

user experience, requirements engineering, software maintenance, and mobile UX

design. She is certified with Certified Professional for Requirements Engineering

(CPRE) and Certified Tester Foundation Level (CTFL). She had experience working

on Fundamental Research Grant and Short-Term Research Grant in collaboration with

other universities. She was also appointed as a technical program committee for sev-

eral international conferences. She held the Program Coordinator position, Final Year

Project Coordinator, became the External Examiner for the Lincoln University Col-

lege, Programme Advisory Committee for Malaysian Polytechnic, and a Panel for

Technical Vocational Education and Training (TVET) workshop.

Nur Iman Mardhiah Hamid is a student at Universiti Kuala Lumpur. She suc-

cessfully delivered her project, which focused on mobile application solutions. Her

research interests include software engineering, software development life cycle, and

software usability evaluation. She attained various kinds of awards, such as first place

in a programming competition, aerial arts beginner level award, and a gold standard

of the Duke of Edinburgh's award. Hence, she portrayed a well-balanced state of mind

in different aspects of life.

Anitawati Mohd Lokman is a professor at Universiti Teknologi MARA (UiTM).

Her expertise is in Kansei Engineering, Emotional User Experience, and Emotion &

Design. She is the pioneer of Kansei Engineering in Malaysia and currently serves as

the president of the Malaysia Association of Kansei Engineering (MAKE) and the

Research Initiative Group for Emotion, Kansei, and Design Engineering (RIG EKDE)

at UiTM.

Article submitted 2020-12-10. Resubmitted 2021-01-25. Final acceptance 2021-03-16. Final version published as submitted by the authors

iJIM ‒ Vol. 15, No. 10, 2021 117