research concepts for HIM (health information management) -

Development of an Instrument to Measure Students’ Perceptions of Information Technology Fluency Skills: Establishing Content Validity

Development of an Instrument to Measure

Students’ Perceptions of Information

Technology Fluency Skills: Establishing

Content Validity

by Marcia Sharp, MBA, RHIA

Introduction

This article describes the process undertaken to develop and validate a tool to measure

students’ perceptions of their information technology (IT) fluency skills. Why is this important?

There is a growing concern that students today are not prepared to live, learn, and work in a

technology-rich society. 1–3

Today’s college students do not have the necessary IT fluency skills

despite their widespread use of the Internet. 4–6

Studies that assess students’ IT fluency skills

show gaps between the perception and reality of these skills. 7, 8

These studies use many

assessments and instruments to evaluate students’ IT fluency skills; however, tools have not been

developed specifically to assess allied health students’ perceptions of their IT fluency skills.

The purpose of this study is to establish the content validity of an instrument to measure

students’ perceptions of their IT fluency skills using a rigorous judgment-quantification process.

The IT fluency instrument developed and validated herein will be used for future studies

comparing allied health students’ perceptions of their IT fluency skills with their actual IT

fluency skills.

Review of the Literature

The assessment of students’ perception of IT fluency skills derives from the National Research

Council’s IT fluency report of 1999. The report challenges the use of the term computer literacy,

which implies just having a particular skill or basic knowledge, whereas fluency involves deep

understanding and critical thinking skills with the ability to adapt to changes in technology. 9 The

Computer Science and Telecommunications Board of the National Research Council (NRC)

devised the concept of fluency with information technology (FITness) to measure the ability of

an individual to handle information technology. While computer literacy is defined with a focus

on computer skills, specifically the ability to use a few computer applications, FITness requires

that people understand information technology well enough to apply it productively in work

situations and in their daily lives, to recognize when information technology may assist or hinder

2 Perspectives in Health Information Management, Summer 2010

the achievement of goals, and to adapt to changes in and the advancement of information

technology. 10, 11

FITness requires three kinds of knowledge: contemporary skills, foundational concepts, and

intellectual capabilities. 12

Contemporary skills, the ability to use today’s computer applications,

enable an individual to apply information technology immediately. 13, 14

Contemporary skills are

an essential component of job readiness. Foundational concepts explain the how and why of

information technology. Foundational concepts are defined as the ability to understand the basic

principles of computers, networks, and information systems. 15, 16

Intellectual capabilities are the

higher-level thinking skills needed to apply information technology in complex and sustained

situations. For instance, the ability to identify where errors exist in a database and solve such

problems requires more than just the ability to enter data into a database. Also, the ability to

understand the changing technology industry allows an intellectually capable individual to

investigate alternatives to antiquated products and processes. Intellectual capabilities empower

people to manipulate the medium to their advantage and to handle unintended and unexpected

problems. 17, 18

Because foundational concepts, intellectual capabilities, and contemporary skills

are essential to the IT fluency concept, they serve as the three constructs from which this tool

was developed. Although many assessment instruments exist to measure students’ IT fluency

skills, no studies have been undertaken in the field of allied health, more specifically health

information management.

Methodology

Overview

Content validity is an essential step in the development of new empirical measuring devices

because it represents a beginning mechanism for linking abstract concepts with observable and

measurable indicators. 19

Lynn (1986) describes content validation as a two-step process

beginning with the developmental stage and ending with the judgment-quantification process. 20

Stage one of the process, the development stage, requires a comprehensive review of the

literature to identify content for the instrument and establish relevant domains. In this study, the

literature review identified approximately 30 to 40 articles on the subject of information

technology fluency. After the literature was reviewed and the items were constructed, the entire

instrument was developed with instructions and scoring guidelines.

The second stage, judgment-quantification, occurs when a panel of experts, working

independently, evaluates the instrument and rates items of relevance according to the content

domain. 21

In addition, item content and clarity, as well as overall instrument comprehensiveness,

are evaluated in this stage. Berk (1990) suggests that expert panel members evaluate how

representative the items are of the content domain. 22

As part of this process, expert panel

members should be asked to provide revisions for items that are not consistent with conceptual

definitions. 23

Clarity of items is another element for content experts to evaluate. 24

Finally, the

instrument should be evaluated, as a whole, for overall comprehensiveness. As Grant and Davis

(1997) note, “This step is necessary because an instrument may have acceptable interrater

agreement, but still not cover the content domain.” 25

When measuring content validity, it is necessary to utilize a quantitative measure, the content

validity index (CVI). 26–28

The CVI is calculated by tallying the results of the expert reviewers.

The degree to which the expert panelists agree on the relevance determines whether the items are

relevant or irrelevant. A Likert-type scale is used to determine relevance. Items that are irrelevant

are scored with a 1, items that are somewhat relevant are scored with a 2, items that are quite

Development of an Instrument to Measure Students’ Perceptions of Information Technology Fluency Skills: Establishing Content Validity

relevant are scored with a 3, and items that are highly relevant are scored with a 4. Only items

scored 3 and 4 are considered relevant and thus are used to calculate the actual CVI.

Instrument

This research required the drafting of a Perceptual IT Fluency Skills Student Survey for use

with allied health students. Before information was gathered, local Institutional Review Board

(IRB) approval was obtained from the University of Memphis and the University of Tennessee

Health Science Center. During the summer and fall of 2009, information on IT fluency and

establishing content validity was gathered. Instruments and information from various sources

were reviewed, and draft survey items were created. The draft survey included measures of

students’ perceptions of their IT fluency skills based on their contemporary skills, foundational

concepts, and intellectual capabilities. The contemporary skills section was composed of eight

multiple choice questions related to the student’s ability to set up a computer, use a word

processor to create a document, use technology to find information, or create a spreadsheet. The

foundational concepts portion contained six multiple-choice questions that focus on the student’s

knowledge of computer operations, networks, and e-mail. The intellectual capabilities section

included five multiple-choice questions to elicit the student’s ability to manage computer

problems, adapt to new technology, and communicate concepts.

Sample

A panel of experts was used to validate the draft Perceptual IT Fluency Skills Student Survey.

The content validation process described by Lynn (1986) was used. 29

A panel of experts

including allied health educators and health information managers was asked to participate in the

process of validating the instrument for content validity to measure allied health students’

perceptions of their IT fluency skills based on the NRC definition of IT fluency. The panel of

experts was selected based on their knowledge of information technology, their chosen

profession within the health information field, and their having at least five years of experience

monitoring and assessing students’ IT fluency skills. The members consisted of individuals from

education and private healthcare entities and were selected because of their involvement in

developing programs for teaching information technology skills to allied health students.

According to Lynn (1986), no more than 10 panel members should be used. 30

This panel

consisted of seven members: three educators and four healthcare professionals. Educators held

the rank of assistant professor or above, and the healthcare professionals were a director of

clinical information systems, a director of health information management, a manager of veteran

services, and a senior systems analyst. All panel members contacted agreed to evaluate the

instrument and provide feedback. All feedback from the experts was received within two months

of initial contact.

Data Collection

A cover letter explaining the purpose of the instrument; literature defining IT fluency concepts

such as contemporary skills, foundational concepts, and intellectual capabilities; and instructions

on how to complete the rating form were e-mailed to the panel of experts in November 2009.

The researcher made a follow-up phone call to the experts to verbally explain the process and to

ensure understanding of the process. The panel was asked to judge the items for clarity,

relevance, and item content using a 1-to-4 scale as described above. The members were asked to

provide suggestions for any revisions or changes needed. The Content Validity Setup designed

by Lynn (1986) was used as a model for this task. 31

After all correspondence was received regarding content validity for each item, a focus group

was held to evaluate the instrument for overall comprehensiveness. Six of the seven panel

4 Perspectives in Health Information Management, Summer 2010

members participated in the focus group. The objective of the focus group was to reach

consensus on the overall comprehensiveness of the instrument, that is, to determine whether the

experts felt the instrument measured what it was intended to measure.

Findings

The calculation or proportion that is sufficient for determining content validity agreement was

explored in the literature. A CVI of 0.70 represents average agreement; 0.80, adequate

agreement; and 0.90, good agreement. 32,33

According to Lynn (1986), when there are six or more

judges, the CVI should be no lower than 0.78 for an item to be judged acceptable. 34

A CVI of

1.00 indicates 100 percent agreement between raters. A CVI was calculated for each item (see

Table 1) and for the overall instrument.

Results from the panel of experts yielded a 0.87 overall content validity index. Six items had a

CVI below 0.78 and were deleted from the instrument. Two experts suggested minor revisions

regarding the clarity or wording of the items, and those revisions were incorporated into the

instrument. One expert suggested that the word “connected” be defined in question 5 under

contemporary skills (this question was subsequently deleted from the survey draft because its

CVI was below 0.78). Another expert suggested that the word “system” be changed to

“application” in question 7 of the contemporary skills category (also subsequently deleted). Once

all items had been evaluated and all changes were made, the revised instrument was sent to the

panel of experts to evaluate the overall instrument.

The focus group discussed the instrument for overall comprehensiveness. None of the experts

suggested additional content or changes at this time. The CVI for the revised instrument (which

can be found in Appendix A) was 1.00. Based on the CVI for each item as well as that for the

overall instrument, it is believed that the instrument contains questions relevant to students’

perceptions of their IT fluency skills.

Conclusion

Content validity is a critical step in the selection and administering of an instrument. The two-

step method used in this study, consisting of a developmental stage and a judgment-

quantification stage, required a comprehensive literature review, item creation, and agreement

from a specific number of experts about the items’ and the entire instrument’s validity. Seven

experts were asked to identify omitted areas and to suggest areas for improvement, and these

revisions were made. The process used to determine content validity proved to offer consistency,

rigor, and structure to the instrument development. High CVI scores were generated for those

items judged relevant to the content domain as well as for the overall instrument. The results

support the content validity of this instrument as a tool for measuring students’ perceived

information technology (IT) fluency skills.

Marcia Sharp, MBA, RHIA, is an assistant professor in the College of Allied Health Sciences at

the University of Tennessee Health Science Center in Memphis, TN.

Development of an Instrument to Measure Students’ Perceptions of Information Technology Fluency Skills: Establishing Content Validity

Notes

1. Dougherty, J., N. Kock, C. Sandas, and R. Aiken. “Teaching the Use of Complex IT in

Specific Domains: Developing, Assessing and Refining a Curriculum Development

Framework.” Education and Information Technologies 7, no. 2 (2002): 137–54.

2. Educational Testing Service. Succeeding in the 21st Century: What Higher Education Must

Do to Address the Gap in Information and Communication Technology Proficiencies. 2003.

Available at http://www.ets.org/ictliteracy/succeeding1.html (accessed October 12, 2009).

3. Salaway, G., and J. B. Caruso. “Students and Information Technology in Higher Education.”

EDUCAUSE Center for Applied Research 6 (2007): 1–124.

4. Hilberg, S. “Fluency with Information and Communication Technology: Assessing

Undergraduate Students.” Doctoral dissertation, Wilmington College, 2007.

5. Katz, I. “Beyond Technical Competence: Literacy in Information and Communication

Technology.” Educational Technology 45, no. 6 (2005): 44–47.

6. Resnick, M. “Rethinking Learning in the Digital Age.” In G. Kirkman (Editor), The Global

Information Technology Report 2001–2002: Readiness for the Networked World. New York:

Oxford University Press, 2002.

7. McEuen, S. F. “How Fluent with Information Technology Are Our Students?” Educause

Quarterly 4 (2001): 8–17. Available at http://www.educause.edu/ir/library/pdf/eqm0140.pdf

(accessed October 18, 2009).

8. Stone, J., and E. Madigan. “Inconsistencies and Disconnects.” Communications of the ACM

50, no. 4 (2007): 76–79.

9. National Research Council Computer Science and Telecommunications Board. Being Fluent

with Information Technology. Washington, DC: National Academy Press, 1999.

10. Lin, H. “Fluency with Information Technology.” Government Information Quarterly 17, no.

1 (2000): 69–76.

11. National Research Council Computer Science and Telecommunications Board. Being Fluent

with Information Technology.

12. Ibid.

13. Dougherty, J., N. Kock, C. Sandas, and R. Aiken. “Teaching the Use of Complex IT in

Specific Domains: Developing, Assessing and Refining a Curriculum Development

Framework.”

14. National Research Council Computer Science and Telecommunications Board. Being Fluent

with Information Technology.

15. Dougherty, J., N. Kock, C. Sandas, and R. Aiken. “Teaching the Use of Complex IT in

Specific Domains: Developing, Assessing and Refining a Curriculum Development

Framework.”

16. National Research Council Computer Science and Telecommunications Board. Being Fluent

with Information Technology.

17. Dougherty, J., N. Kock, C. Sandas, and R. Aiken. “Teaching the Use of Complex IT in

Specific Domains: Developing, Assessing and Refining a Curriculum Development

Framework.”

18. National Research Council Computer Science and Telecommunications Board. Being Fluent

with Information Technology.

19. Wynd, C. A., B. A. Schmidt, and M. A. Schaefer. “Two Quantitative Approaches for

Estimating Content Validity.” Western Journal of Nursing Research 25, no. 5 (2003): 508–

18.

6 Perspectives in Health Information Management, Summer 2010

20. Lynn, M. “Determination and Quantification of Content Validity.” Nursing Research 35

(1986): 382–85.

21. Ibid.

22. Berk, R. “Importance of Expert Judgment in Content-Related Validity Evidence.” Western

Journal of Nursing Research 12 (1990): 659–71.

23. Lynn, M. “Determination and Quantification of Content Validity.”

24. DeVillis, R. F. Scale Development: Theory and Applications. Newbury Park, CA: Sage,

1991.

25. Grant, J. S., and L. Davis. “Selection and Use of Content Experts for Instrument

Development.” Research in Nursing & Health 20 (1997): 271.

26. Anders, R. L., J. S. Tomai, R. M. Clute, and T. Olson. “Development of a Scientifically

Valid Coordinated Care Path.” Journal of Nursing Administration 27 (1997): 45–52.

27. Summers, S. “Establishing the Reliability and Validity of a New Instrument: Pilot Testing.”

Journal of Post Anesthesia Nursing 8 (1993): 124–27.

28. Wynd, C. A., B. A. Schmidt, and M. A. Schaefer. “Two Quantitative Approaches for

Estimating Content Validity.”

29. Lynn, M. “Determination and Quantification of Content Validity.”

30. Ibid.

31. Ibid.

32. Ibid.

33. Wynd, C. A., B. A. Schmidt, and M. A. Schaefer. “Two Quantitative Approaches for

Estimating Content Validity.”

34. Lynn, M. “Determination and Quantification of Content Validity.”

Development of an Instrument to Measure Students’ Perceptions of Information Technology Fluency Skills: Establishing Content Validity

Table 1

Content Validity Index (CVI) of Survey Items

Item No.

Rater

1

Rater

2

Rater

3

Rater

4

Rater

5

Rater

6

Rater

7 CVI Contemporary

Skills

1 X X X 0 0 X X 0.71

2 X X X X X 0 X 0.86

3 X X X X X X X 1.00

4 X X X 0 X X X 0.86

5 X X X 0 0 0 X 0.57

6 X X X 0 X X X 0.86

7 X X X 0 0 X X 0.71

8 X X X X X X X 1.00

Foundational

Concepts

9 X X X X X 0 X 0.86

10 X X X 0 X X X 0.86

11 X X X 0 X X X 0.86

12 X X X 0 0 X X 0.86

13 X X X 0 0 X 0 0.57

14 X X X 0 0 X X 0.71

Intellectual

Capabilities

15 X X X X X X X 1.00

16 X X X X X X X 1.00

17 X X X X X X X 1.00

18 X X X X X X X 1.00

19 X X X X X X X 1.00

20 X X X X X X X 1.00

21 X X X 0 0 0 0 0.43

22 X X X X X X X 1.00

23 X X X 0 X X X 0.86

24 X X X X X X X 1.00

25 X X X X X X X 1.00

26 X X X X X X X 1.00

27 X X X 0 X X X 0.86

28 X X X X X X X 1.00

29 X X X 0 X X X 0.86

Overall CVI 0.87

Note: X indicates items of relevance.

8 Perspectives in Health Information Management, Summer 2010

Appendix A

Perceptual IT Fluency Skills Student Survey: Final Draft for Expert Panel Evaluation of Content

Relevance

Statement Content Relevance (please circle or highlight

your choice)

Contemporary Skills: ability to use today’s

computer applications, enable people to

apply information technology immediately.

Participants will answer these questions as

no knowledge, some knowledge, average

knowledge, expert knowledge.

1. When it comes to using basic operating

system features, I consider myself to have

_____.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

2. When it comes to using a software program

to create a text document, I consider myself to

have _____.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

3. When it comes to using a graphics or art

package to create illustrations, slides, or image-

based expression of ideas, I consider myself to

have _____.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

4. When it comes to using a spreadsheet to

model simple processes of financial tables, I

consider myself to have _____.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

5. When it comes to using instructional

materials to learn how to use a new application

or features, I consider myself to have _____.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Foundational concepts: basic principles and

ideas of computers, networks, and

information. Participants will answer these

questions as strongly disagree, disagree,

neutral, agree, strongly agree.

1. I can explain how a computer operates.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Development of an Instrument to Measure Students’ Perceptions of Information Technology Fluency Skills: Establishing Content Validity

2. I can identify a computer hardware problem.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

3. I can identify a computer software problem.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3= quite relevant but needs minor revisions

4 = highly relevant

4. I can define computer storage.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Intellectual capabilities: the ability to apply

information technology in complex and

sustained situations . . . fosters more abstract

thinking about information and its

manipulation. Participants will answer these

questions as strongly disagree, disagree,

neutral, agree, strongly agree.

1. If something went wrong with my computer

or a computer I was using, I would likely:

Ignore the problem

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Troubleshoot the problem myself

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Find a way to work around the problem

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Use online support and/or knowledge bases to

solve the problem

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Use printed reference manuals to identify and

solve the problem

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Ask a friend or family member for help

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

2. I can easily learn new software applications.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

10 Perspectives in Health Information Management, Summer 2010

3. I feel comfortable and confident when using

new technologies.

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

4. When I want to use a new function or feature

in a software application, I would likely:

Suggestions for changes:

Use the application help screens

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Read the user manual

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Call a help desk attendee

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Ask a friend or family member

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Access online resources

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant

Figure it out myself

Suggestions for changes:

1 = not relevant

2 = somewhat relevant

3 = quite relevant but needs minor revisions

4 = highly relevant