Reporting the Results of a Website Evaluation

Abdullah Albelehy

2

Website content

The main (home) page contains the school logo, which is positioned on top left corner of

the page. The top right corner has four hyperlinked icons, from left to right, Calendar,

Lunch meal, School tools, and Rocket news respectively.

Directly under the logo and the icons is the tool bar with five options spreading from left

to right. The options are Schools, District, Staff, Departments, and Contact. These options

contain most of the information in the website. Below chart shows the main categories

under each option. At a glance, School option contains information about primary and

secondary schools. District has many information, which includes information about the

district, athletics details, board of directors and information for Parents/ Students/

Community. Employees resources and tools are under Staff option. Departments option

has details about many departments in the school. Last are the contact information and

Contact options.

Seven flash photos come after the tool bar. Photos are auto rotating, with control arrows

on both sides. Photos are for some primary school kids, secondary school students, school

buildings, and staff (with family members).

Following the photos are three vertical pars. Two of them contains district news, and the

third is Quick links (repeated links, for faster and easier access).

At the bottom of the website there are some links (some of the options in the tool bar) for

direct and easy access. At the right bottom corner, there is the school superintendent contact

info.

3

Techniques employed

Photos, flat icons, colors, and text are used in this website. News are updated continuously,

but again it use text and photos. No movies, animations, sound or interactive techniques

are used.

METHODS

The course facilitator provided two tools to be used in web evaluation. Objective of

evaluation is to identify the strengths and weaknesses of DeRuyter Central School’s

website. Tools are the Content Validity Scale, CVS, (Arnone & Small, 1999) and the

Website Motivational Analysis Checklist (WebMAC) – Professional Version (Small &

Arnone, 2004).

Evaluation was done by author of this report. To eliminate any bias that could happen,

another two evaluators were asked to use the same tools to evaluate the website. Scores

used are the average of the three evaluations.

Home

Schools

Primary

Secondary

District

About

Athletics

BoE

P/S/C

Staff

Facility

Login

Learning plan

Purchase request

Tech.

Mail login

Dep.

Counceling

Fine Arts

Media

Health

Tech.

Transp.

Contact

Contact inf

News Q. Links

4

Instruments Used

The Content Validity Scale

The ten questions in this scale aimed to check and test creditability, information relevance,

typographical and spelling errors, whether the content is biased or not, and contact

information. Scaling is from 1 to 5. 1 represents Strongly Disagree, and 5 represents

Strongly Agree.

The Website Motivational Analysis Checklist (WebMAC) – Professional Version

WebMAC is longer than CVS and has 32 questions, grouped in a specific way to measure

four different criterions which are: 1. How Stimulating the website it is, 2. How Meaningful

the website it is, 3. How Organized the website it is, 4. How Easy-to-use it is.

Scaling is from 0 to 3. 0 represents Strongly Disagree and 3 represents Strongly Agree.

Some questions might not be relevant, and hence can’t be rated, therefore “Not Applicable”

is there. These NA items will be assigned scores, 0 if the site would have benefit from it, 1

if the site didn’t require that item, and 2 if the site was better off for not having that item.

The tool presented a visual representation so that evaluator can clearly see the results. Who

doesn’t want visual representation instead of bare figures and tables!

Description of Evaluators

As mentioned previously, the website is evaluated by three different evaluators, to

overcome and eliminate any possible bias in scores. Two evaluators, plus the report author,

used the two tools and reported their scores.

The first evaluator is a software developer at ONB bank in Khartoum, Sudan. His

experience is over seven years and has rich technical knowledge in web design,

programming, network and database administration.

The second evaluator is a master student in Inclusive Education, at School of Education,

Syracuse university. She is a disability rights advocate, and worked in Kenya for many

years advancing equal rights for people with disability. She is an active member in Young

African Leaders Initiative (YALI).

Important Note: Evaluator from Sudan couldn’t open the website from Sudan. A message

“The requested URL was rejected by the F5 load balancer. Please contact the help desk at

5

(315) 433-8345.” was shown. Two sides, the positive one is that there is a contact phone (I

would rather prefer an email. Who wants to make an international call!). Negative one is

that the website didn’t work from Sudan as well as France!). Can’t tell what is the problem,

but my friend needed to use his technical knowledge to get the website open for him!

RESULTS

Content Validity Score

The table in appendix A shows the evaluation scores for the three evaluators followed by

the average scores (Scale from 1 to 5. 1 represents Strongly Disagree, and 5 represents

Strongly Agree). Final result or total is 44.7 out of 50.

Generally, CVS scores of the website are high and reflect good impression on evaluators.

Below are the frequencies of scores.

Website Motivational Analysis Checklist Score

The tables in appendix B shows the average evaluation scores (Scale from 0 to 3. 0

represents Strongly Disagree, and 3 represents Strongly Agree). In summary, here are the

scores for S, M, O, and E respectively.

0 1 2 3 4 5

Scores

FREQUESNCY OF SCORES

5 4.7 4.3 4

6

Stimulation = 14.0 (8 < S ≤ 16) Needs some improvement

Meaningfulness = 18.3 (16 < M ≤ 24) Highly motivating

Organization = 19.0 (16 < O ≤ 24) Highly motivating

Ease of use = 17.3 (16 < E ≤ 24) Highly motivating

The two overall motivational quality scores:

Value Dimension (V) = S + M = 14 + 18.3 = 32.3

Expectation for Success (XS) = O + E = 19 + 17.3 = 36.3

0

5

10

15

20

25

S M O E

Sc o

re s

WebMAC categories

Scores…

7

INTERPRETATION AND SYNTHESIS OF RESULTS

Content Validity Scale

The average score of the Content Validity Scale is 44.7 out of 50 (≈ 90%), which is high

and reflect very good status. Evaluators strongly agreed that the website is free of linguistic

errors and updated continuously. Other scores vary between 4.0 and 4.7 which is good. In

summary, DeRuyter Central School website has a very good credibility, free of errors, up

to date and unbiased.

WebMAC Professional

How Stimulating?

Evaluators gave the website 14 points out of 24. As a result, the website falls in the

“NEEDS SOME IMPROVEMENT” zone. No surprise as the website has no eye-catching

videos, sounds, or photos. There is no (or very limited) interactivity, and website visitors

can only browse, read, and contact via phone or email. Photos are overstretched and have

a poor quality. Students can browse the website for specific information, and most likely

will leave soon. Nothing to glue them so they can spend long time.

High Expectation for Success

Low Expectation for Success

Low

Value

High Value

X 32.3V;36.3X S

8

On the other hand, the website has a good overall look. All or most of information are one

to two clicks away.

How Meaningful?

Meaningfulness of website receives 18.3 out of 24. Website is at the bottom of the

“HIGHLY MOTIVATING” category. Evaluators believes that information presented are

credible, browsing is easy due to clear menus and accurate. Seems website admin is

continuously updating the website and making sure to keep it accurate and unbiased.

Unfortunately, evaluators think that the website doesn’t provide links to relevant websites

and has no or limited ability for users and browsers to actively participate and contribute

ideas.

How Organized?

Organization seems to be the strength point of this website and located in the “HIGHLY

MOTIVATED” category. Score is relatively high with 19 points out of 24. All evaluators

gave full scores to the clarity of the website’s purpose, as well as language correctness and

free of grammatical errors. Other scores are generally high with one exception. While

browsing, evaluators had hard time getting back to the Home page.

How Easy to Use?

Ease of use got 17.3 out of 24. Like Meaningfulness, Ease of Use is at the bottom of the

“HIGHLY MOTIVATING” category. Navigation of the website is super easy. Everything

is in front of you, with simple and nice organization. Loading of website is also fast and no

noticeable delays at all.

Unfortunately, evaluators were disappointed due to the lack of control and passiveness they

experienced while checking the website. Photos were also a source of unhappiness as they

are not crisp, vivid and clear. Everyone has a 4K resolution screen these days!

Overall Value and Expectation for Success

The website has a high overall motivational quality scores. Value dimension and

Expectation for success scores are 32.2 and 36.3 respectively. These scores places the

9

website into the “Awesome Website” category, which means this site falls into the Average

to High range for Value and the average-to-high range for Expectation of Success.

RECOMMENDATIONS/SUGGESTIONS FOR IMPROVEMENT

The website is generally rated Very Good. It has relevant and useful information to its

audiences. Browsing is an easy task and doesn’t need previous knowledge or experience,

thanks to the website’s well-designed structure. Last and not least, the website isn’t left

with outdated information, but regularly updated.

On the other side, there are many areas of improvement. Below are some suggestions:

▪ Photos need to be replaced with higher resolution ones. Most of current devices have

high definition or even 4K resolution displays. The website must take advantage of this

advancement in technology. Loading time need to be considered while choosing new

photos so they will not affect or make the website slower.

▪ There is no Home button or icon. If a user dive in the website, there is no way to go

back to the home/main page. The only way is to rewrite the link in the browser. Home

button is very important and should be accessed from anywhere in the website.

▪ Website is very passive and there is no (or very limited) interaction between school’s

website and targeted audiences. The website need to adopt some interactive techniques.

Blog and/or chatting zone are few examples to mention.

▪ Email addresses are embedded and hyperlinked in shorter names. Unfortunately

pressing these short names doesn’t take you to any email client as it should do, and

email addresses aren’t explicitly mentioned so users can simply copy/paste in their

email clients or browsers. I recommend that the website developers explicitly write the

email addresses, and hyperlink them at the same time. Doing so will enable users to

either copy/paste the email addresses in their email client or click them to automatically

open an email client.

▪ Videos to be added. No one wants to read, people prefer watching short videos than

reading a couple of pages. The videos can be YouTube ones, and hyperlinked in the

website.

▪ Few months ago, Mark Zuckerberg, Facebook founder, announced that they reach 2

billion subscribers. The School website must have a page there, as well as other social

10

media websites. The most important issue is the content management, and to interact

with people out there.

REFLECTION

According to http://www.internetlivestats.com there are 1.26 billion websites now (Sep.

2017). With such number, creditability, correctness, interactivity, free of linguistic errors,

contact information, etc. are serious and critical issues. A need for objective and free of

bias tool/s is a must.

For me, this is the first time to objectively and accurately measure such qualities (in my

opinion, three evaluators aren’t enough to get accurate results, but accepted for this study

and report). When I explored the website for the first time, before checking the tools, I

thought the website is almost perfect, except the quality of the photos. After doing the

evaluation, I can see that there are much more issues than the quality of the photos.

For the creditability of information sources, it was difficult to prove it. It wasn’t easy to

check every link and make sure if it works fine or not, yet very important. Despite these

difficulties, reviewing the website from different angels (i.e. creditability, ease of use, etc.)

was good and pleasant experience, as I used to focus on visuals only.

Gained knowledge and experience from this task will be of great help in future if I should

do any evaluation/s, and I can tell that I need to do so. The experience isn’t only helpful

while evaluating websites, but also while building/creating websites. For any website that

I will build, these qualities will be present and will be taken care of.

Hopping that provided suggestions can be implemented and reflected, and I’m sure they

will positively and greatly enhance the website. Also, I strongly recommend another

evaluation round after applying these suggestions. I prefer different evaluators (but same

tools) to eliminate any presumed knowledge or bias, but scores must be compared to see

the progress.

11

REFERENCES

http://www.deruytercentral.org/

Content Validity Scale (Arnone & Small, 1999)

Website motivation analysis checklist (WebMAC) Professional (v 2.0) - (Arnone & Small,

2004)

12

APPENDIXES

APPENDIX A

CVS scores

Statements Eval.

1

Eval.

2

Eval.

3 Ave.

1 The source of information for this Web site is

credible 5 5 4 4.7

2 There is a way to contact the author of this Web site

if necessary. 5 5 3 4.3

3 The factual information or the content of the Web

site seems accurate. 5 4 5 4.7

4

If the Web site presents concepts or principles in its

domain (e.g. science, art), they are appropriately

presented without confusing or missing information.

4 4 5 4.3

5

There are no typographical or spelling errors that

could potentially cause the information at this Web

site to be misunderstood.

5 5 5 5.0

6 The content is appropriate for the intended

audience. 5 4 4 4.3

7 The links from this site appear to be credible. 5 4 4 4.3

8 The Web site appears to be free of bias. 4 4 4 4.0

9 The information at this Web site is current enough

for the type of information it includes. 5 5 5 5.0

10 The links form this Web site appear current and

unbiased. 4 4 4 4.0

TOTAL 47 44 43 44.7

13

APPENDIX B

Website Motivational Analysis Checklist Score

S

1. 2.0

5. 1.7

9. 1.3

13. 2.0

17. 2.7

21. 1.0

25. 1.3

29. 2.0

Total S 14.0

M

2. 3.0

6. 1.3

10. 3.0

14. 2.7

18. 2.3

22. 2.7

26. 0.7

30. 2.7

Total M 18.3

O

3. 2.0

7. 3.0

11. 2.7

15. 2.3

19. 3.0

23. 3.0

27. 2.3

31. 0.7

Total O 19.0

E

4. 3.0

8. 2.0

12. 2.3

16. 1.3

20. 2.3

24. 0.7

28. 2.7

32. 3.0

Total E 17.3

14

APPENDIX C

Content Validity Scale© (Arnone & Small, 1999)

Instructions: Rate your agreement with the following statements.

1. The source of information for this Web site is credible.

Strongly Disagree 1 2 3 4 5 Strongly Agree

2. There is a way to contact the author of this Web site, if necessary.

Strongly Disagree 1 2 3 4 5 Strongly Agree

3. The factual information or content of the Web site seems accurate.

Strongly Disagree 1 2 3 4 5 Strongly Agree

4. If the Web site presents concepts or principles in its domain (e.g. science, art), they are appropriately presented without confusing or missing information.

Strongly Disagree 1 2 3 4 5 Strongly Agree

5. There are no typographical or spelling errors that could potentially cause the information at this Web site to be misunderstood.

Strongly Disagree 1 2 3 4 5 Strongly Agree

6. The content is appropriate for the intended audience.

Strongly Disagree 1 2 3 4 5 Strongly Agree

7. The links from this site appear to be credible.

Strongly Disagree 1 2 3 4 5 Strongly Agree

8. This Web site appears to be free of bias.

Strongly Disagree 1 2 3 4 5 Strongly Agree

9. The information at this Web site is current enough for the type of information it includes.

Strongly Disagree 1 2 3 4 5 Strongly Agree

10. The links from this Web site appear current and unbiased.

Strongly Disagree 1 2 3 4 5 Strongly Agree

15

APPENDIX D

Website Motivational Analysis Checklist (WebMAC)

Professional (v. 2.0)

DIRECTIONS:

WebMAC Professional was designed for professional educators to assess Web sites for use

in classroom instruction or for student use in homework assignments or projects. Before

using WebMAC Professional, it's a good idea to spend some time exploring the Web site to

be evaluated in order to have some familiarity with its content and structure. You may need

to go through the Web site at least once more to complete this checklist.

Rate your level of agreement with each of the 32 items on the next two pages by placing

the appropriate number value on the line in front of each item. If you are not sure about any

item, select the best response you can give.

3 = strongly agree

2 = somewhat agree

1 = somewhat disagree

0 = strongly disagree

NA = Not Applicable

Example of completed item:

3 0. This Web site makes me happy.

After you have rated all 32 items, go back and look at each NA item. You should place one of

the following scores next to the NA for that item.

0 = the Web site would have benefited if it had included the item (e.g. audio).

1 = the Web site didn't require that item.

2 = the Web site was better off for not including that item.

Copyright  1999 (revised 2004) by Ruth V. Small & Marilyn P. Arnone, Syracuse, New

York. All rights reserved. No part of the material protected by this copyright notice may be

reproduced or utilized in any form or by any means, electronic or mechanical, including

photocopying, recording , or by any information storage and retrieval system (except for

non-profit educational use), without the written permission of the copyright owner.

16

WebMAC Professional (v. 2.0)

Your Name

______________________________________________________________

_______________

Grade level/subject

______________________________________________________________

________

Web Site URL

______________________________________________________________

____________

Directions: Place the appropriate number about this Web site on the line preceding

each item.

3 = strongly agree

2 = somewhat agree

1 = somewhat disagree

0 = strongly disagree

NA = Not Applicable

___ 1. The screen layout of this Web site is attractive.

___ 2. There is a menu or site map at the beginning that describes what content

is contained within the Web site.

___ 3. Visual (e.g. videos, photographs) or audio information included in this Web

site helps to clarify or describe the topic.

___ 4. Navigating this Web site does not require any special skills or experience

by me or my students.

___ 5. There is an eye-catching title and/or visual on the home page of this Web

site that will attract students' attention.

17

___ 6. This Web site provides valuable links to other useful Web sites.

___ 7. The purpose of this Web site is always clear to me.

___ 8. The Web site has a help function that I or my students can use at any

time.

___ 9. This Web site provides opportunities for interactivity.

___ 10. The Web site's information is provided by credible sources.

___ 11. The directions for using this Web site are simple and clear.

___ 12. This Web site makes it easy to search or query for information.

___ 13. The information on my topic included in this Web site is interesting.

___ 14. The information contained in this Web site is current and up-to-date.

___ 15. There is useful information on my topic at this Web site.

___ 16. This Web site's graphics are crisp and clearly visible.

___ 17. The variety of formats (e.g. text, photos, sound) helps to maintain

attention.

___ 18. The information at this Web site is accurate and unbiased.

___ 19. All of the information at this Web site is presented using clear and

consistent language and style.

___ 20. All of this Web site's hyperlinks are active and fully functioning.

18

WebMAC Professional (v. 2.0)

page 2

Name__________________________URL _____________________________

3 = strongly agree

2 = somewhat agree

1 = somewhat disagree

0 = strongly disagree

NA = Not Applicable

___ 21. This Web site has novel or unique features that make it more interesting

for students.

___ 22. There is little or no unimportant or redundant information at this Web

site.

___ 23. The text at this Web site is well-written without grammatical, spelling,

or other errors.

___ 24. At all times, I can control what information at this Web site I wish to

see or show my students.

___ 25. This Web site stimulates curiosity and exploration.

___ 26. This Web site provides opportunities for users to actively participate or

contribute ideas.

___ 27. This Web site provides the appropriate amount of information on the

topic for a lesson or assignment.

19

___ 28. All buttons and other navigation mechanisms for moving around at this

Web site work the way they should.

___ 29. The colors and/or background patterns used in this Web site are

pleasing.

___ 30. This Web site provides opportunities to communicate with its authors.

___ 31. No matter where I am in this Web site, I can return to the home page.

___ 32. The amount of time it takes for pictures, games, videos, etc. to appear

on the screen at this Web site is reasonable for teaching/learning

situations.

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_ _ __ _ _ _ _ _ _ _ _ _ _ _ _

There is enough at this Web site to warrant a return visit. YES NO

I would recommend this Web site to my colleagues. YES NO

What is the greatest strength of this Web site?

______________________________________________________________

______________________________________________________________

What needs improvement at this Web site?

______________________________________________________________

______________________________________________________________

______________________________________________________________

______________________________________________________________

20

Scoring WebMAC Professional (v. 2.0)

Before transferring scores, go back to each item designated as "NA" and choose a score for

that item based on the following criteria:

• 0 points if you feel that the site would have benefited if it had included the item (e.g. audio).

• 1 point if you feel that the Web site didn't require that item.

• 2 points if you believe the site was better off for not including that item. When all 32 items have a number score, transfer each score into the appropriately

numbered space in one of the four columns below. Then add the scores for each column and

write the total on the appropriate line.

S M O E

1. ___ 2. ___ 3. ___ 4. ___

5. ___ 6. ___ 7. ___ 8. ___

9. ___ 10. ___ 11. ___ 12. ___

13. ___ 14. ___ 15. ___ 16. ___

17. ___ 18. ___ 19. ___ 20. ___

21. ___ 22. ___ 23. ___ 24. ___

25. ___ 26. ___ 27. ___ 28. ___

29. ___ 30. ___ 31. ___ 32. ___

TOTAL S ___ TOTAL M ___ TOTAL O ___ TOTAL E ___

The S column reflects how STIMULATING this Web site is for you.

The M column reflects how MEANINGFUL this Web site is to you.

The O column reflects how ORGANIZED you think this Web site is.

The E column reflects how EASY-TO-USE this Web site is for you.

21

Transforming Scores

Now you may transform your scores into visual representations so that you can

clearly see the results of your evaluation.

1. Take each of the four total scores above and plot them (either with a dot or a bar)

on the graph below.

2. If you use dots, connect the dots to make a line.

Now take the total scores from the four categories (S, M, O, E) and add them together in

the following way to get your two overall motivational quality scores.

S + M = ______ (V) O + E = ______ (XS)

The V score reflects a summary motivation score on the Value dimension; i.e.

how stimulating and meaningful the Web site is. The XS score reflects a summary

motivation score on the Expectation for Success dimension; i.e. how organized and

easy-to-use the Web site is.

HIGHLY MOTIVATING

S M O E

30 24

2 16

8

0

NEEDS MUCH IMPROVEMENT

NEEDS SOME IMPROVEMENT

22

Plotting the Scores

To plot scores on the scoring grid,

1. Place a dot for the V score along the Value continuum and a dot for the XS score

along the Expectation for Success continuum on the grid.

2. Draw straight lines to their intersection point, representing the overall motivational

quality score of the Web site

In the example below the evaluated Web site received a Value score of 40 and

an Expectation for Success score of 18. Their intersection point falls within the lower

right quadrant of the grid. Now go to the next page to record your scores.

Example Completed Scoring Grid

To record your actual scores onto the scoring grid.

1. Plot the score for V along the Value continuum and the score for XS along the

Expectation for Success continuum.

2. Draw straight lines to the intersection point.

High Expectation for Success

Low Expectation for Success

Low Value High Value

X 40V;18XS

23

| | | | | | | | | | | | | | | | | | |

| | | |

0

48

0 48 32 16

16

32

Average to high for value/Average to

high for expectation for success

24 40

40

8

8

Average to high for value/Below average to

low for expectation for success

Below average to low for value/Below

average to low for expectation for success

Awesome

Website!

High Expectation for Success

Low Expectation for Success

Low Value High Value

Average to high for expectation for

success/Below average to low for value