AN EXAMINATION OF DEMONSTRABLE CLASSROOM BEHAVIORS OF
DIGITALLY LITERATE EDUCATORS: A DELPHI STUDY
CHAPTER 1. INTRODUCTION
“Recent years have yielded significant advances in computing and communication
technologies, with profound impacts on society. Technology is transforming the way we
work, play, and interact with others” (National Academy of
Sciences, 2017, p. 1). Technology has impacted almost all areas of modern American life.
“It has already eliminated and created jobs, but more frequently it has transformed jobs
and the way they are performed” (National Academy of Sciences, 2017, p. 52).
Technology has affected the personal lives of those living in modern society as
communications, shopping, and social interactions increasingly occur online.
“Technology has transformed employment by automating some tasks and creating
the need for new ones, a trend that is likely to continue” (National Academy of Sciences,
2017, p. 79). Today, most jobs require some level of technical competency, which
requires the general workforce to possess some level of fluency with computers and
information technology (National Academy of Sciences, 2017).
According to the Pew Research Center report, “The Future of Jobs and Jobs
Training,” skills necessary for the modern workforce include the following:
•process-oriented and system-oriented thinking;
•journalistic skills, including research, evaluation of multiple sources, writing,
and speaking;
•understanding algorithms, computational thinking, networking, and
programming;
•grasping law and policy;
•an evidence-based way of looking at the world;
•time management;
•conflict resolution;
•decision-making;
•locating information in the flood of data;
•storytelling using data; and
•influencing and consensus building. (Rainie & Anderson, 2017, p. 14)
Information literacy and communicating across global contexts and in technology-
infused environments were also listed as necessary skills in the Pew report (Rainie &
Anderson, 2017). “Many of the new skills necessary for jobs of the future require digital
skills to be successful” (Rainie & Anderson, 2017, p. 32).
The World Economic Forum’s Future of Jobs report described the world at the
beginning of the Fourth Industrial Revolution, requiring new skill sets in part because
65% of children entering school in 2017 will work in jobs that currently do not exist.
(Freeman, Adams Becker, Cummins, Davis, & Hall Giesinger, 2017). “Developing skills
that enable learners to use computers to gather data, break it down into smaller parts, and
analyze patterns will be an increasing necessity to succeed in our digital world. While
coding is one aspect of this idea, even those not pursuing computer science jobs will need
these skills to work with their future colleagues” (Freeman et al., 2017, p. 4).
The increasing digitization of the workplace will require schools to adapt to this
change. The education system exists in part to prepare students to be productive members
of society. “Because education will significantly determine the success of the United
States in responding to the changing workplace, a better understanding of effective
strategies is critical” (National Academy of Sciences, 2017, p. 12).
Beth Corzo-Duchardt, assistant professor at Muhlenberg College, noted in the
Pew report, “the fast pace of technological innovation means that any educational
program that successfully trains workers to succeed in future jobs must focus on
fundamentals like critical thinking, self-directed learning, basic computer literacy and, in
some fields, basic math, science, and writing skills” (Rainie & Anderson, 2017, p. 72).
Introduction to the Problem
Current students enrolled in American elementary, middle, and high schools were
practically born with technology in their hands. Their digital footprints were often created
while in utero by well-meaning parents posting sonogram images on social media. These
students are adept at making personal communications via the Internet, yet they typically
are not proficient in the application of digital literacy skills in an academic setting (Li &
Ranieri, 2010; Stout, 2017).
The amount of information online surpassed one zettabyte at the end of 2016
(CISCO, 2017), or the equivalent to over 250 billion DVDs (Arthur, 2011). Students need
to be able to use information critically—to “evaluate resources carefully and determine
how to use relevant information to solve problems and make wise decisions” (Farmer,
2011, p. 387). The need to adequately evaluate information is mostly going unmet by
public schools (Alexander, Adams Becker, & Cummins, 2016; Johnson, 2007; Lyons,
2010). Furthermore, “there is considerable variance in digital literacy policies and
implementation programs across the US and the rest of the world” (Alexander et al.,
2016, p. 1).
During the 2016 Presidential election, the term fake news brought the issue of
false information on the internet to the public eye (Rogers & Bromwich, 2016). Fake
news is defined as “news articles that are intentionally and verifiably false, and could
mislead readers” (Alcott & Gentzkow, 2017, p. 213). Scores of false websites and
accounts introduced misinformation to social media feeds during the Presidential
campaign and were subsequently discussed by legitimate news outlets and academics
(Alcott & Gentzkow, 2017).
The spotlight on misinformation and source evaluation demonstrated a pressing
need for digital literacy education. In “Social Media and Fake News in the 2016
Election,” researchers Alcott and Gentzkow (2017) stated, “in the presence of motivated
reasoning, education gives people better tools to counterargue against incongruent
information” (p. 228).
A study conducted by the Stanford History Education Group highlighted why fake
news is problematic. More than 7,800 students from 12 states were asked to perform
several tasks on the internet such as differentiating news from advertisements on a web
site. More than 80% of the middle schoolers tested believed an item with the label
“sponsored content” was a legitimate news article instead of an advertisement (Stanford
History Education Group, 2016). Students were also asked to ascertain the credibility of
photographs posted online. Nearly 40% of the high school students evaluated decided a
photograph provided strong evidence, regardless of the source. “Overall, young people’s
ability to reason about the information on the Internet can be summed up in one word:
bleak” (Stanford History Education Group, 2016, p. 4).
The fake news events of the 2016 presidential election, especially when
considered with the results of the 2016 Stanford study, demonstrated why digital
citizenship and digital literacy skills were important to modern digital society. Digital
citizenship was defined as “the norms of appropriate, responsible behavior with regard to
technology use” (Ribble & Bailey, 2004, p. 13). Digital literacy was defined as “the
ability to use information and communication technologies to find, understand, evaluate,
create, and communicate digital information, an ability that requires both cognitive and
technical skills” (American Library Association [ALA], 2013, p. 2).
Improving digital literacy skills was included in the 2017 Horizon Report from
the New Media Consortium and Consortium for School Networking as one of the six
most significant challenges facing technology integration in schools today. Both higher
education and the contemporary workforce call for digital savants who can seamlessly
work with different media and new technologies as they emerge. A major element of
fostering this fluency is recognizing that simply understanding how to use a device or
certain software is not enough; teachers, staff, and students must be able to make
connections between the tools and the intended outcomes, leveraging technology in
creative ways that allow stakeholders to more intuitively adapt from one context to
another. Ownership of this movement must be shared and supported among school
leaders and practitioners because digital fluency is an important thread that runs through
practically every facet of teaching and learning. (Freeman et al., 2017, p. 9)
The need for digital literacy and digital citizenship education is reflected in
federal legislation and standards. The Children’s Internet Protection Act (CIPA) of 2000
mandated school districts provide students with education in digital citizenship. Digital
Citizenship instruction was also a component of the International Society for Technology
in Education National Educational Technology (ISTE) Standards in both 2007 and 2016.
It was also mentioned in the Every Student Succeeds Act (ESSA) of 2015.
Ribble and Bailey (2004) defined nine components of digital citizenship—digital
access, digital commerce, digital communication, digital literacy, digital etiquette, digital
law, digital rights and responsibilities, digital health and wellness, and digital security. Of
those nine tenets, much of the current digital citizenship instruction offered to students
emphasizes safety and security topics such as creating strong passwords and
cyberbullying avoidance (Stout, 2017; Suppo, 2013).
In order to effectively educate students in digital citizenship, “a significant
amount of conceptual and evaluation work is needed to ensure that its goals are
welldefined and its outcomes successfully achieved” (Jones & Mitchell, 2015, p. 2).
Lindsay and Davis (2010) likened students online with no education in digital citizenship
to sheep being let out of the gate to wander aimlessly.
According to Poore (2011), there were six components to digital literacy—
accessing information, managing information, evaluating information, creating new
understandings, communicating with others, and using information and communication
technologies appropriately. Application of digital literacy skills encompassed
“troubleshooting abilities, purposes for connecting, skills in using popular technology
tools, as well as communication literacy, and web literacy” (Blummer, 2008, p. 38).
Several similar terms, such as digital fluency and digital competence, were found in
research pertaining to this subject.
The Merriam-Webster Dictionary defined citizenship as “the status of being a
citizen; membership in a community; the quality of an individual’s response to
membership in a community” (Citizenship, n.d.). In contrast, the Merriam-Webster
Dictionary defined literacy as “the quality or state of being literate” (Literacy, n.d.).
Literate was defined as “educated, cultured; able to read and write; …having knowledge
or competence” (Literate, n.d.).
Digital citizenship describes the actions of a participant in a digital environment.
Digital literacy is having competence or knowledge of the digital environment. It is
possible to be a citizen of a community without having a high level of competence or
knowledge about the community. Being literate about an environment implies a level of
knowledge higher than that of a participant. For this reason, the current study focused
primarily on the more advanced skillset required for digital literacy, particularly the
internet literacy or web literacy components of digital literacy.
Digital literacy is considered “a social, political, economic, and cultural product,
and has significant implications for current education, culture, society, and community
development in the digital age” (Lee, 2014, p. 30). Digital literacy is the extension and
application of skills learned in digital citizenship instruction. While students have grown
up online, in current digital citizenship education, they are not necessarily being provided
the skills needed to use digital information productively, especially in an academic
environment (Li & Ranieri, 2010; Stout, 2017).
Research indicates a lack of digital literacy competency in teachers may be
exacerbating the problem (Berardi, 2015; Hollandsworth, Dowdy, & Donovan, 2011).
Digital literacy “requires a very specific set of educator knowledge and teaching skills
compared to other goals currently set under the digital citizenship umbrella” (Jones &
Mitchell, 2015, p. 2). The “disparity in digital literacy levels appears particularly among
older and less-educated people and women” (Lee, 2014, p. 31). In a field where the
average practitioner age is 42.4 (National Center for Education Statistics, 2012), it is
understandable the typical classroom teacher may not feel a high level of competence
with digital literacy instruction. The lack of confidence may be due to a lack of training
or education in teaching digital literacy either in a teacher preparation program or in a
formal professional development experience (Farmer, 2011). Hobbs (2010) identified
some additional reasons for teachers not creating instruction in the use of digital literacy
skills such as an unfamiliarity of the technology, a lack of understanding of effective
technology integration, or prioritizing high stakes testing superseding 21st Century skills
such as digital literacy. The current study attempts to identify essential behaviors of
digitally literate teachers so the identified behaviors can be used as a model for
professional learning for all classroom teachers. While digital citizenship education
provides the foundation for student success in a networked world, digital literacy is the
application of higher order thinking skills in that network. For the current study, the focus
is on digital literacy education, particularly as it pertains to internet literacy.
Background of the Study
Digital Citizenship
Ribble, in 2006, completed a dissertation in which his goal was to research,
develop, test, and validate a handbook on digital citizenship for technology leaders.
During the course of his work, Ribble identified nine key themes of digital citizenship
education. He also is credited with coining the phrase and with the first definition of the
term. The working definition of digital citizenship for the purpose of the current study is
“the norms of behavior with regard to technology use” (Ribble, 2006, p. 1). The nine
themes of digital citizenship education indicated by Ribble were etiquette,
communication, education, access, commerce, responsibility, rights, safety, and security.
Digital citizenship has alternately been defined as “appropriate technology usage” and
“making safe, responsible, respectful choices online” (Common Sense Media, 2018). For
the purposes of the current study, the original definition crafted by Ribble is used.
Li and Ranieri (2010) conducted a study in which the digital competency of
Chinese students was measured using the Instant Digital Competence Assessment created
by the University of Florence. Digital competence has often been used interchangeably
with the term digital literacy. Students overall rated “pass” on the competence
assessment, which is an average score, rather than “good” or “excellent.” Students scored
at the average level regardless of factors such as having a computer or Internet access at
home, or the frequency of computer and Internet use. The findings further supported the
assertion that students are in need of instructional interventions by educators in regards to
digital literacy skills.
One of the criticisms of digital citizenship instruction in schools is it focuses on
low level skills rather than higher-order thinking and problem solving (Oxley, 2010).
Oxley (2010) asserted students must understand not only the rules themselves but the
rationale behind the rules and “to be able to make thoughtful and critical decisions when
confronted by opportunities to engage in inappropriate and irresponsible online behavior”
(p. 1). Alexander et al. (2016) further stated, “there is still work to be done to develop
programs and curricula that provide learners with more hands-on, deeper experiences” (p.
1).
A 2010 analysis of covariance (ANCOVA) study on student normative behavior
after being exposed to digital citizenship instruction supported this assertion. Boyle
(2010) found there was no change in normative behavior for the themes of security and
access. Where there was normative behavior, there was no demonstration of “actual”
behavior or skills. The findings supported the assertion stating what is learned in
traditional digital citizenship instruction is not being transferred to student practice
online. In contrast, digital literacy, which focuses more on the application of digital
citizenship skills when consuming and creating content with technology, is how students
can apply critical thinking and problem-solving skills.
Digital Literacy
Digital literacy, in addition to being one of the nine components of digital
citizenship, is also a freestanding collection of skills and competencies. The definition of
digital literacy “includes a range of abilities from basic computing skills to the creation of
multimodal texts” (Blummer, 2008, p. 38). Blummer (2008) identified five components
of digital literacy—computing, troubleshooting, fluency with common applications,
communications literacy, and web literacy.
A New Media Consortium Horizon survey of more than 450 education leaders,
faculty, and staff conducted in 2016 found educators do not have a clear understanding or
shared definition of digital literacy, which impedes the ability to instruct students on
digital literacy skills. Among the recommendations from researchers is the need for
institutions to “experiment further with digital literacy pedagogy, curriculum, and
program design–and share their results broadly” (Alexander et al., 2016, p. 4).
In contrast to the Boyle (2010) study, which found there was no skills transference
to behavior online after exposure to digital citizenship instruction, Lee (2014) found
students did transfer skills to online behavior after digital literacy instruction. Lee (2014)
conducted a study of 145 individuals seeking assistance at a community center in the
Southern region of the United States. The study included pre- and post-surveys and tests
of individuals receiving instruction on four types of basic skills—operational skills, usage
skills, communication and interaction skills, and creation skills—designed to increase the
operational digital literacy competence of subjects. Not surprisingly, the instruction
resulted in increased performance on the post-tests, but participants also indicated on the
surveys they expanded their application of these basic skills to pursue jobs and interests
online, connect with others, and assist other technology users.
Internet Literacy
Internet literacy, also referred to as web literacy, is “the ability to access,
understand, critique, and create information and communication content online”
(Livingstone, 2008, p. 102). It is listed as one of the components of digital literacy
(Blummer, 2008).
Miller and Bartlett (2012) conducted a survey of 509 elementary and secondary
teachers in the United Kingdom pertaining to internet literacy as a component of digital
literacy. The study determined 79.8% of the teachers found digital fluency was “very
important” for students to possess (p. 47), yet over half of the teachers reported regularly
noticing a lack of digital literacy in their students. These teachers rated their students as
“below average” in internet literacy skills (p. 48). The only skill rated above average was
an understanding of how search engines operate.
November (2012) stated educators must prepare students to make meaning from
information online and guide their ability to create and publish information worldwide
using what he termed “web literacy.” November (2012) further asserted students
possessing these skills are often self-taught. November and Mull (2012) identified three
pillars to web literacy. These pillars were purposeful search, effective organization and
collaboration, and sharing and making sense of information. Leowy (2016) further
elaborated on the definition of web literacy as “the ability to explore, create, and connect
online” (p. 91).
Johnson (2007) asserted, “functional internet literacy cannot be conceptualized
simply in terms of basic knowledge skills” (p. 436) and identified several categories of
common online activities and the technical skills required to conduct those activities.
These categories included communication, information, recreation, and technical
activities.
Statement of the Problem
Students are learning the basics of being safe online but they are rarely taught to
think and problem-solve online. Digital citizenship instruction mainly occurs at the
lowest common denominator—basic skills. Education in digital literacy, or the
application of digital citizenship skills, has been found to be inconsistent. (Berardi, 2015;
Hollandsworth et al., 2011).
Aspects of digital literacy education are not explicitly taught to teachers in
preservice or in-service trainings, which further exacerbates the issue. Teachers are left to
fend for themselves should they choose to develop competence in teaching digital literacy
(Ribble, 2006). Few districts offer professional learning on topics such as
•choosing the right tool to find, use, or create information;
•finding a person online, for example an expert in a certain discipline, and
establishing their contact details;
•establishing who owns information and ideas found online;
•using advanced search options to limit and refine searches; and
•assessing whether an online resource or person is credible and trustworthy.
Furthermore, administrators have no tool to use to evaluate the digital literacy
competency of classroom teachers. In many cases, they use the colloquial definition of “I
know it when I see it” to identify effective digital literacy practices demonstrated by
educators. The current study attempts to identify the observable behaviors of teachers
who are competent in digital literacy instruction. These behaviors, as identified by an
expert panel, were used to generate a definition of a digitally-literate teacher and a
nonevaluative walk-through document to guide decisions concerning professional
development.
Purpose of the Study
The purpose of the current study was twofold. The first purpose was to utilize a
panel of expert practitioners to identify the characteristics of a teacher with a high level of
digital literacy competency due to the lack of a widely agreed-upon definition.
The second purpose of the current study was to examine observable teacher
behaviors integral to teaching digital literacy in order to create a non-evaluative
walkthrough document for measuring the digital literacy proficiency of teachers. Data
from these walk-throughs may be used by administrators or instructional technology staff
to guide professional development decisions. The current study primarily benefits campus
and central office administrators and instructional technology personnel.
Using a Delphi method study, expert educators in the digital literacy field
generated a list of observable teacher behaviors exemplifying a high level of competence
in digital literacy instruction. This information was formatted into a definition of a
digitally literate teacher and a brief, non-evaluative walk-through document. The current
study was accomplished in four iterations, the first of which identified the teacher
behaviors, the second and third rounds focused on achieving consensus on the items to be
included on the definition and walk-through document, the fourth round allowed panelists
to provide input on the definition and walk-through document.
Rationale
Much of the research on digital literacy focuses on student behaviors and skills
acquisition (Blummer, 2008). There appears to be a gap in the literature describing
demonstrable behaviors of teachers possessing a high competence of digital literacy.
There is also a gap in the literature about professional development for teachers, other
than self-paced online courses, to hone digital literacy skills. Blummer (2008) conducted
a review of literature pertaining to digital literacy instruction and concluded, “additional
research is required to reveal the applicability of these findings…this remains especially
true for the self-reported teacher classroom inquiries” (p. 44). Similarly, additional
research in digital literacy has been advised in the findings of other studies (Hatlevik,
Ottestad, & Throndsent, 2014; Jones & Mitchell, 2015; Suppo, 2013).
Numerous assessments to measure student digital literacy skills exist, such as the
Northstar Digital Literacy Assessment (Northstar Digital Literacy Project, 2017), the
Atomic Learning Digital Literacy Assessment (Hoonuit Professional Learning, 2017),
and the Instant Digital Competence Assessment (Calvini, Cartelli, Fini, & Ranieri, 2008).
There is not a widely-used tool identifying teacher proficiency in teaching digital literacy
skills to students. The current study addressed this deficiency by focusing research on the
behaviors digitally literate teachers exhibit in the classroom.
The current study also created a definition of a digitally literate teacher and an
informal walk-though form for administrators or instructional technology personnel. The
definition and walk-through form may be used to guide professional learning decisions to
address skills gaps and improve instruction.
Research Questions
“In many schools, despite significant investment in technology, teachers are not
making effective use of the engaging instructional practices of digital and media literacy”
(Hobbs, 2010, p. 25). The reasons for the oversite may include teachers having a lack of
familiarity with technology tools or effective integration techniques. The current study
created a definition of a digitally literate teacher and a non-evaluative document
identifying possible areas of digital literacy deficiency in teachers and areas at which
teachers excel. The research questions this study addressed are as follows:
Research Question 1 (RQ1) What are the indicators of digital literacy competency in
teachers as identified by a panel of experts?
Research Question 2 (RQ2) Which of these behaviors could be observed on a
walkthrough observation?
Research Question 3 (RQ3) What are the elements of such a form?
Significance of the Study
Suppo (2013) specifically addressed the need for “a tool to measure the level of
digital citizenship demonstrated daily by administrators, teachers, and students” (p. 96) as
a conclusion of his study. Hatlevik et al. (2014) recommend studies “to describe good
teacher roles for fostering students’ digital competence” (p. 229). The current study
addressed the gap in the literature as described in these two studies as well as those
identified by Jones and Mitchell (2015) and Blummer (2008).
The current study could have applications in any K-12 environment to aid
administrators in identifying the level of digital literacy competency in teachers.
Application of the definition and the subsequent walk-through document developed
during the current study will serve to identify skills a teacher may need to develop.
Teachers may use the items on the walk-through document as a self-assessment as they
develop curricula and assessments for students to ensure digital literacy is addressed in
their instruction and assignments. The definition may be used during the hiring process to
identify desired skills for teachers. The walk-through form may also be used to guide
administrators in determining appropriate professional development opportunities.
The results of the current study may provide valuable information for educators
and administrators in developing and sustaining effective digital literacy practices among
teachers. The identification of essential observable behaviors among students and
teachers can serve as a guide for professional development in technology integration,
curriculum design, and formative assessment creation.
Definition of Terms
The following terms clarify the various types of literacies that are interwoven to
compile the necessary skills students must have for networked, global participation.
Digital citizenship is the foundational set of skills upon which the other literacies are
based.
Digital citizenship
Digital citizenship is “the norms of behavior with regard to technology use”
(Ribble & Bailey, 2004, p. 13). As defined by Ribble and Bailey (2004), there are nine
tenets of digital citizenship education: digital access, digital commerce, digital
communication, digital literacy, digital etiquette, digital law, digital rights and
responsibilities, digital health and wellness, and digital security.
Digital competencies
Digital competencies “determine what students know and understand about the
virtual worlds and how they use the tools of technology” (Zhao, 2009, p. 177). This term
is often used interchangeably with digital literacy.
Digital literacy
Digital literacy is “the ability to use information and communication technologies
to find, evaluate, create, and communicate information, requiring both cognitive and
technical skills” (ALA, 2012). The aspect of digital literacy used primarily for the
purposes of the current study is internet literacy or web literacy.
Information and communications technologies (ICT)
Information and communications technologies (ICT) are a “diverse set of
technological tools and resources used to communicate, and to create, disseminate, store,
and manage information” (Blurton, 1999, p. 1). Examples of ICT include computers,
tablets, cellular phones and networks, and the internet.
Information literacy
Information literacy is defined as “the ability to search for, select, critically
evaluate and use information for solving problems in various contexts, such as
independent project work in schools” (Limberg, Sundin, & Talja, 2013, p. 96).
Internet literacy
Internet literacy is “the ability to access, understand, critique, and create
information and communication content online” (Livingstone, 2008, p. 102). The term
also refers to web literacy.
Likert-type scale
A Likert-type scale is a measurement device used primarily in psychological or
social science environments used to determine attitudes, values, and opinions.
Participants complete a questionnaire indicating the extent to which they agree or
disagree with a number of statements (Likert, 1932).
Media literacy
Media literacy is “the process of critically analyzing and learning to create one’s
own messages in print, audio, video, and multimedia. Its emphasis is on the learning and
teaching of these skills through using mass media texts in primarily school-based
contexts” (Hobbs, 1998, p. 1). The term has also come to include media on the internet as
well. It is often bundled with digital literacy to create the term digital media literacy to
refer to media found on the internet.
Technology integration
Technology integration refers to “using computers effectively and efficiently in
the general content areas to allow students to learn how to apply computer skills in
meaningful ways” (Dockstader, 1999, p. 73).
Walk-through
A walk-through is defined as a “brief, structured, nonevaluative classroom
observation by the principal that is followed by a conversation between the principal and
the teacher about what was observed” (The Center for Comprehensive School Reform
and Improvement, 2007, p. 1).
Web literacy
November (1998) coined the phrase “web literacy” and identified three pillars:
purposeful search; effective organization and collaboration; sharing and making use of
information. (p. 2). Web literacy is the same as internet literacy.
Assumptions and Limitations
The researcher assumed the experts chosen to participate in the current study were
able to document expertise in their content and profession. The current study relied on
self-reported experience of participants and/or on the existence of published materials
authored or co-authored by the participants as a partial measure of expertise.
The researcher assumed teacher behaviors demonstrating competence in digital
literacy instruction can be observed and that the statements made by chosen experts were
accurate and truthful.
The researcher assumed the data was limited due to the number of participants.
Due to time limitations and the scope of the project, the current study was limited to the
K-12 educational environment and may or may not have applications to university
students and professors.
The current study was limited to teachers and librarians conducting direct, faceto-face
instruction.
The current study was limited in that it did not address the effectiveness of digital
citizenship or digital literacy instruction. The current study also did not evaluate current
digital citizenship or digital literacy curricula. Nor did the current study determine
whether or not instruction has a lasting impact on student digital behaviors.
Nature of the Study
The nature of the current study was a Delphi study of experts in the digital literacy
field, particularly as it pertained to internet literacy. The Delphi method attempted to
“obtain the most reliable consensus of opinion of a group of experts” (Dalkey & Helmer,
1963, p. 458). In order to obtain the most reliable consensus, expert identities remained
anonymous to prevent panelists from influencing one another. The expert panelists then
completed a series of four rounds of questionnaires and surveys. Information and answers
to the questions were relayed to participants to allow for adjustments in subsequent
rounds.
The Delphi method “derives its importance from the realization that projections
into the future… are largely based on the personal expectations of individuals rather than
predictions derived from a well-established theory” (Helmer, 1967, p. 4). The Delphi
process has been used for various applications such as program planning, needs
assessment, policy refinement, and resource allocation (Hsu & Sanford, 2007).
Aspects of the Delphi method include the questions all being centered around one
particular problem; inquiry into the reasoning behind the experts’ answers to the initial
question, which is often open-ended; and considering relevant factors and what type of
data would be necessary in studying those factors, which will in turn provide a superior
answer to the central question (Dalkey & Helmer, 1963). The current convergence of
opinion is well-suited to applying real-world knowledge to a particular topic for the
purpose of goal setting, policy investigation, or predicting the occurrence of future events
(Hsu & Sandford, 2007).
Using the Delphi method in the current study, a list of observable teacher
behaviors exemplifying a high level of competence in digital literacy instruction was then
formatted into a brief, non-evaluative walk-through document and a definition of a
digitally literate teacher. The process required four iterations, the first of which identified
characteristics and behaviors of a digitally literate teacher; panelists in the second and
third rated those behaviors using a Likert-type scale. Internal consistency of the second
and third round questionnaires was determined using a Cronbach’s alpha statistical
analysis. The fourth round consisted of participant input on the sample document created
from the responses from Rounds 2 and 3.
Organization of the Remainder of the Study
The current study is organized into five chapters. Chapter 1 includes the statement
of the problem, background and need for the study, research questions, significance,
definitions of terms, assumptions, limitations, and a general overview of remaining
chapters. Chapter 2 provides a review of relevant literature regarding digital citizenship,
literacies related to education in a digital environment, educational technology standards,
technology integration, classroom walk-throughs, Likert-type scales, and the Delphi
method. Chapter 3 includes a restatement of the purpose, the research design, setting,
participants, and descriptions of data collection, analysis, and verification. Chapter 4
includes an introduction, the summary of the data collection methods, analysis of the
data, a discussion of the data, and key findings. Chapter 5 provides a summary of the
study findings, conclusions, recommendations for future research, implications for action,
and a closing statement.
CHAPTER 2. LITERATURE REVIEW
There are a wide variety of definitions for the terms included in the research
concerning digital citizenship and digital literacy. The variety may be due to the relative
newness of the field. The resulting ambiguity can lead to confusion and frustration to
educators as they work through how to best educate students for a digital world. Student
growth in technology access usage is “outpacing the preparedness for digital guidance as
shared by educators” (Berardi, 2015, p. 12).
The lack of precision as to the terms was also evidenced in the multiple strategies
and lists of skills various researchers and practitioners purported to be relevant to digital
citizenship and digital literacy instruction. The current study first examined these
ambiguities and attempted to provide some clarity in identifying demonstrable behaviors
that teachers with a high level of digital competence exhibit.
Standards
The International Society for Technology in Education (ISTE) first published the
National Educational Technology Standards (NETS) in 1998. These standards included
basic operations and concepts: standards that addressed technology tools for productivity,
communications, research, and problem-solving and decision-making.
Standards for Students
In the first round of the ISTE standards for students, digital citizenship and digital
literacy were addressed in the standard for social, ethical, and human issues. The goal was
that “students understand the ethical, cultural, and societal issues related to technology”
and “[s]tudents practice responsible use of technology systems, information, and
software” (ISTE, 1998, p. 14).
Revised standards were released by ISTE in 2007. The second iteration of
standards included headings for digital citizenship as well as research and information
fluency. The digital citizenship standard stated, “[s]tudents understand human, cultural,
and societal issues related to technology and practice legal and ethical behavior” (ISTE,
2007, p. 2). Students demonstrate digital citizenship by advocating and practicing “safe,
legal, and responsible use of information and technology” (ISTE, 2007, p. 2). Students
also demonstrate personal responsibility and leadership relating to digital citizenship.
The research and information fluency standard promoted that “students apply
digital tools to gather, evaluate, and use information” (ISTE, 2007, p. 1). The standard
elaborated that students “locate, organize, analyze, evaluate, synthesize, and ethically use
information from a variety of sources and media” and “evaluate and select information
sources and digital tools based on the appropriateness to specific tasks” (ISTE, 2007, p.
1).
In 2016, ISTE released updated standards named the ISTE Standards for Students.
Similar to the 2007 standards, these new standards included a heading titled “digital
citizen.” The goal was that “students recognize the rights, responsibilities and
opportunities of living, learning and working in an interconnected digital world, and they
act and model in ways that are safe, legal, and ethical” (ISTE, 2016, p. 1). Students
should “engage in positive safe, legal and ethical behavior when using technology,
including social interactions online or when using networked devices” (ISTE, 2016, p. 1).
The third version of standards also included components for students to manage their
digital identity, demonstrate an understanding of intellectual property, and manage data to
promote digital privacy and security.
The research and information fluency standard from 2007 was rewritten in 2016
under the heading “knowledge constructor.” Students were to demonstrate the ability to
“critically curate a variety of resources using digital tools to construct knowledge,
produce creative artifacts and make meaningful learning experiences for themselves and
others” (ISTE, 2016, p. 1). While doing so, students were charged with not only finding
information, but also evaluating the “accuracy, perspective, credibility and relevance of
information, media, data or other resources” (ISTE, 2016, p. 1). A comparison of the
evolution of the ISTE standards for students can be found in Tables 1 and 2.
Standards for Educators
Corresponding standards for teachers were initially released in 2000. A standard
for Social, Ethical, Legal, and Human Issues was included in the document. “Teachers
understand the social, ethical, legal, and human issues surrounding the use of technology
in PK-12 schools and apply that understanding in practice” (ISTE, 2000, p. 9).
Performance indicators for the standard included modeling and teaching “legal and
ethical practice related to technology use” (ISTE, 2000, p. 9). Other digital
citizenshiprelated indicators included safe use of resources and providing access to all
students. The second-generation ISTE Standards for Teachers were published in 2008.
Teachers were to “promote and model digital citizenship and responsibility” (ISTE,
2008, p. 2). Teachers “understand local and global societal issues and responsibilities in
an evolving digital culture and exhibit legal and ethical behavior in their professional
practice” (p. 2). Indicators for the standard included “advocate, model, and teach safe,
legal and ethical use of digital information and technology, including respect for
copyright, intellectual property, and the appropriate documentation of sources” (p. 2).
Teachers should also “promote and model digital etiquette and responsible social
interactions related to the use of technology and information” (p. 2).
Table 1
Summary of Digital Citizenship Skills from Various Iterations of International Society for
Technology in Education (ISTE) Student Standards
ISTE (1998) ISTE (2007) ISTE (2016)
Students understand the ethical,
cultural, and societal issues
related to technology.
Students practice responsible use
of technology systems,
information, and software
Students understand human,
cultural, and societal issues
related to technology and
practice legal and ethical
behavior.
a. Advocate and practice
safe, legal, and responsible use
of information and technology
b. Exhibit a positive attitude
toward using technology that
supports collaboration,
learning, and productivity
c. Demonstrate personal
responsibility for lifelong
learning
d. Exhibit leadership for
digital citizenship
Students recognize the rights,
responsibilities and opportunities
of living, learning and working
in an interconnected digital
world, and they act and model in
ways that are safe, legal, and
ethical. Students:
a. cultivate and manage
their digital identity and
reputation and are aware of
the permanence of their
actions in the digital world.
b. engage in positive, safe,
legal and ethical behavior
when using technology,
including social interactions
online or when using
networked devices.
c. demonstrate an
understanding of and respect
for the rights and obligations
of using and sharing
intellectual property.
d. manage their personal
data to maintain digital
privacy and security and are
aware of datacollection
technology used to track their
navigation online.
Note: Adapted from National Educational Technology Standards for Students (1998) by the
International Society for Technology in Education; ISTE Standards for Students (2007) by the
International Society for Technology in Education; and 2016 ISTE Standards for Students (2016)
by the International Society for Technology in Education.
Table 2
Summary of Information Literacy Skills from Various Iterations of International Society
for Technology in Education (ISTE) Student Standards
ISTE (1998) ISTE (2007) ISTE (2016)
Students use a variety of media
and formats to communicate
information and ideas effectively
to multiple audiences.
Students use technology to
locate, evaluate, and collect
information from a variety of
sources.
Students evaluate and select new
information resources and
technological innovations based
on the appropriateness for
specific tasks.
Students apply digital tools to
gather, evaluate, and use
information.
a. Plan strategies to guide
inquiry
b. Locate, organize,
analyze, evaluate, synthesize,
and ethically use information
from a variety of sources and
media
c. Evaluate and select
information sources and
digital tools based on the
appropriateness to specific
tasks
d. Process data and report
results
Students critically curate a
variety of resources using digital
tools to construct knowledge,
produce creative artifacts and
make meaningful learning
experiences for themselves and
others. Students:
a. plan and employ
effective research strategies
to locate information and
other resources for their
intellectual or creative
pursuits.
b. evaluate the accuracy,
perspective, credibility and
relevance of information,
media, data or other
resources.
c. curate information from
digital resources using a
variety of tools and methods
to create
collections of artifacts that
demonstrate meaningful
connections or
conclusions.
d. build knowledge by
actively exploring real-world
issues and problems,
developing ideas and
theories, and pursuing
answers and solutions.
Note: Adapted from National Educational Technology Standards for Students (1998) by the
International Society for Technology in Education; ISTE Standards for Students (2007) by the
International Society for Technology in Education; and 2016 ISTE Standards for Students (2016)
by the International Society for Technology in Education.
The revised NETS, renamed the ISTE Standards for Educators, were released in
2017 to correspond with the 2016 student standards. The latest standards emphasize the
“transformation of the digital learning landscape” (ISTE, 2017, p. 1) and outline three
major shifts in thinking about technology teacher standards. When the ISTE NETS were
first released, focus was on how to use technology tools. The 2008 revision of these
standards emphasize how these tools should be used for instruction and student learning
(ISTE, 2017). The latest iteration of technology standards include “innovation, disruption
and evolution” (p. 2).
The shift in focus from “technology skills” to “learning with technology” became
“transforming learning with technology” (ISTE, 2017, p. 2). Correspondingly, pedagogy
shifted throughout the three versions of standards from “teacher-driven” to
“studentcentered” to “student-driven” as students were empowered to take control of their
learning (p. 2).
The 2017 ISTE Standards for Educators described the “empowered professional”
as a learner, a leader, a citizen, a collaborator, a designer, a facilitator, and an analyst.
The “Citizen” standard is as follows:
Educators inspire students to positively contribute and responsibly participate in
the digital world. Educators:
a. Create experiences for learners to make positive, socially responsible
contributions and exhibit empathetic behavior online that build relationships
and community.
b. Establish a learning culture that promotes curiosity and critical examination of
online resources and fosters digital literacy and media fluency.
c. Mentor students in the safe, ethical and legal practice with digital tools and
protection of intellectual rights and property.
d. Model and promote management of personal data and digital identity and
protect student data privacy. (ISTE, 2017, p. 4)
A comparison of the three versions of teacher standards for digital citizenship and
information literacy can be found in Table 3.
Table 3
Summary of Digital Citizenship Skills from Various Iterations of International Society for
Technology in Education (ISTE) Educator Standards
ISTE (2000) ISTE (2008) ISTE (2017)
Social, Ethical, Legal, and
Human Issues
Teachers understand the social,
ethical, legal, and human issues
surrounding the use of
technology in PK–12 schools
and apply that understanding in
practice. Teachers:
a. Model and teach legal
and ethical practice related to
technology use.
b. Apply technology
resources to enable and
empower learners with diverse
backgrounds, characteristics,
and abilities.
c. Identify and use
technology resources that
affirm diversity.
d. Promote safe and healthy
use of technology resources.
e. Facilitate equitable
access to technology resources
for all students.
Promote and Model Digital
Citizenship and Responsibility
Teachers understand local and
global societal issues and
responsibilities in an evolving
digital culture and exhibit legal
and ethical behavior in their
professional practices.
a. Advocate, model, and
teach safe, legal, and ethical
use of digital information and
technology, including respect
for copyright, intellectual
property, and the appropriate
documentation of sources.
b. Address the diverse
needs of all learners by using
learnercentered strategies
providing equitable access to
appropriate digital tools and
resources.
c. Promote and model
digital etiquette and
responsible social interactions
related to the use of
technology and information.
d. Develop and model
cultural understanding and
global awareness by engaging
with colleagues and students
of other cultures using digital
age communication and
collaboration tools.
Citizen
Educators inspire students to
positively contribute and
responsibly participate in the
digital world. Educators:
a. Create experiences for
learners to make positive,
socially responsible
contributions and exhibit
empathetic behavior online
that build relationships and
community.
b. Establish a learning
culture that promotes
curiosity and critical
examination of online
resources and fosters digital
literacy and media fluency.
c. Mentor students in the
safe, ethical and legal
practice with digital tools
and protection of intellectual
rights and property.
d. Model and promote
management of personal
data and digital identity and
protect student data privacy.
Note: Adapted from National Educational Technology Standards for Teachers (2000) by the
International Society for Technology in Education; ISTE Standards for Teachers (2008) by the
International Society for Technology in Education; and ISTE Standards for Educator (2017) by
the International Society for Technology in Education.
Digital Citizenship
Ribble and Bailey (2004) first coined the term digital citizenship and defined it as
“the norms of appropriate, responsible behavior with regard to technology use” (p. 1). It
is currently the definition most commonly accepted in K-12 education. Digital citizenship
was identified as having nine themes: digital etiquette, digital communication, digital
education, digital access, digital commerce, digital responsibility, digital rights, digital
safety, and digital security. These themes were later revised by Ribble (2006) to digital
etiquette, digital communication, digital literacy, digital access, digital commerce, digital
law, digital rights and responsibilities, digital health and wellness, and digital security.
In 2006, Ribble stated educators had not remained current with technology
advances, which was linked to students not knowing appropriate and inappropriate uses
of technology. Digital citizenship was “organized to provide structure to a digital society”
(p. 29). The purpose of digital citizenship instruction was to create a dialog for ethical
technology use and is an “attempt to provide a consistent message to all technology users
of what they need to do to be productive users of digital technologies” (p. 72). Ribble
(2006) also maintained digital citizenship education for students should be embedded into
their education in a way so as to promote students to appropriately use technology in their
lives outside the classroom as well as citizens of a digital world.
Models of Digital Citizenship
Farmer (2011) built on the work of Ribble and Bailey to expand the definition of
digital citizenship as “the ability to use technology safely, responsibly, critically,
productively, and civically” (p. 388). In addition to learning appropriate use, digital
citizenship education represents “the need for learners to learn coping skills and
demonstrate that they can contribute to the digital society” (p. 388).
In order to do this, teachers must integrate digital citizenship across the
curriculum and add several components to instruction. Farmer (2011) outlined a
framework for adding informational practices to digital citizenship instruction. These
practices included awareness, engagement, manipulation, and application of information.
In the awareness phase, learners recognize situations where information is needed to
solve a problem. During engagement, learners learn how to access, decode, and evaluate
information. The learner also determines if information is relevant to the immediate
problem or task and their rights and responsibilities regarding information in the
engagement phase. Transforming digital information into knowledge occurs during the
manipulation phase through “interpretation, organization, synthesis, reformatting,
changing, relating or combining it with other information” (Farmer, 2011, p. 390).
Manipulation of digital information requires four skills that are learned during this phase:
extracting information, deciding how to represent information, determining method of
information manipulation, and knowing how to do the manipulation (Farmer, 2011). The
last phase, application, is when the learner determines what to do with the information
and determine which actions he/she will take or recommend as a result of the
information.
Common Sense Media is a nonprofit organization dedicated to educating students,
parents, and educators about how to help “kids thrive in a world of media and
technology” (Common Sense Media, 2018). Based on research conducted by Howard
Gardner as part of the Good Play Project at the Harvard Graduate School of Education,
Common Sense Media developed a K-12 scope and sequence of digital citizenship lesson
plans. The topics of these lessons are divided into the following topics:
•Creative Credit and Copyright;
•Cyberbullying and Digital Drama;
•Digital Footprints and Reputation;
•Information Literacy; • Internet Safety;
•Privacy and Security;
•Relationships and Communication; and
•Self-image and Identity.
The Good Play Project is “focused on five ethical fault-lines” that are “ethically
salient in new media environments” (Harvard Project Zero, 2017). The five themes are
identity, credibility, privacy, participation, and ownership/authorship.
Searson, Hancock, Soheil, and Shepherd (2015), as part of the EduSummIT 2013
digital citizenship working group, attempted to place digital citizenship in a global
context. The group explored if a “locus of digital citizenship even exists” because there is
such a lack of consensus about what constitutes digital citizenship (Searson et al., 2015,
p. 730). Searson et al. (2015) defined digital citizenship as “the characteristics of an
individual’s behavior, especially within collaborative environments, when engaged with
digital tools, such as computers, mobile devices, cell phones and tablets” (p. 730). The
group examined two models, the ISTE model from the ISTE and the iKeepSafe model by
the nonprofit iKeepSafe organization. Both were found by the working group to be well
respected by the educational community and recommended for K-12 organizations
seeking implementation options for digital citizenship. Searson et al. (2015) labeled this
model “aspirational” because it contained characteristics that are “universal ideals” (p.
732).
The ISTE model identified the following characteristics of a positive digital
citizen:
1. Equal digital rights and access for all.
2. Treating others with respect in online environments; no cyber-bullying.
3. No stealing or damaging others’ digital work, identity or property.
4. Appropriate decisions when communicating through digital channels.
5. Using digital tools to advance learning and keeping up with changing
technologies.
6. Responsible online purchasing decisions while protecting payment
information.
7. Upholding basic digital rights in digital forums.
8. Protecting personal information from forces that might cause harm.
9. Limiting physical and psychological health risks of technology (Brichacek,
2014).
The iKeepSafe model created the “BEaPRO Index” of competencies and skills
necessary for successful digital citizenship:
•Balance: Balancing Digital Usage;
•Ethics: Practicing Ethical Digital Usage;
•Privacy: Protecting Personal Information;
•Relationships: Maintaining Healthy & Safe Relationships;
•Reputation: Building a Positive Reputation; and
•Online Security: Achieving Digital Security.
iKeepSafe program developers recommended both national and site-based
committees be created to address “challenges and opportunities that digital citizenship
technology presents” (Searson et al., 2015, p. 735). Table 4 demonstrates how these three
models compare to the original nine elements of digital citizenship as identified by
Ribble, Bailey, & Ross (2004).
Table 4
Comparison of Various Digital Citizenship Models
Ribble (2009) ISTE (2016) iKeepSafe (2011) Common Sense
Media (2018)
Digital etiquette Treating others with
respect in online
environments
Building a positive
reputation
Digital footprint and
reputation
Digital
communication
Appropriate
decisions when
communicating
through digital
channels
Protecting personal
information
Relationships and
communication
Digital literacy Using digital tools to
advance learning
and keeping up with
changing
technologies
Balancing digital
usage
Information literacy
Digital access Equal digital rights
and access for all
X X
Digital commerce Responsible online
purchasing
decisions while
protecting payment
information
X Internet safety
Digital law No stealing or
damaging others’
digital work, identity
or property
Practicing ethical
digital usage
Creative credit and
copyright
Table 4 (continued)
Ribble (2009) ISTE (2016) iKeepSafe (2011) Common Sense
Media (2018)
Digital rights and
responsibilities
Upholding
basic digital
rights in digital
forums
X
Cyberbullying and
digital drama
Digital health and
wellness
Limiting physical
and psychological
health risks of
technology
Maintaining healthy
and safe
relationships
Self-image and
identity
Digital security Protecting personal
information from
forces that might
cause harm
Achieving digital
security
Privacy and security
Note: Adapted from Nine Themes of Digital Citizenship by Mike Ribble (2009);
Citizenship in the Digital Age (2014) by the International Society for Technology in
Education, BEaPROTM Digital Citizenship Toolkit by iKeepSafe (2011); and Scope &
Sequence: Common Sense K-12 Digital Citizenship Curriculum by Common Sense
Education (2014).
Hollandsworth et al. (2011) defined a digital citizen as someone in the digital
world “able to get an education, work, buy, sell, trade, interact with others, and be
entertained; all of the same characteristics of a traditional society” (p. 40). The authors
conducted a survey of over 500 library media specialists on their perceptions of digital
citizenship awareness at their schools. Forty-nine percent of the respondents believed
teachers were aware of digital citizenship issues. Forty-two percent of the group reported
that teachers were “mostly unaware” of the issues (Hollandsworth et al., 2011, p. 45). The
respondents generally believed a curriculum of digital citizenship would be a necessary
beginning, however, they believed it would take a collaborative effort of all educators,
parents, and community to “effectively teach and consistently practice the tenets of digital
citizenship” (Hollandsworth et al., 2011, p. 44).
Studies on Digital Citizenship
Based on the results of their survey, Hollandsworth et al. (2011) recommended
parents begin teaching children digital citizenship from the first time the child touches a
device. Recommendations for schools included beginning comprehensive digital
citizenship education in Kindergarten including “peer mentor programs, effective role
models, educational faculty/staff awareness, enhanced awareness of the risks, and most
importantly, a proactive versus reactive approach” (Hollandsworth et al., 2011, p. 39).
Berardi (2015) conducted a study of 64 elementary teachers to ascertain their
perceptions of value and self-efficacy in teaching digital citizenship. The study
determined 96% of the participants either agreed or strongly agreed that there was a need
for elementary schools to teach digital citizenship (Berardi, 2015). Of the various aspects
of digital citizenship, 69% of the teachers believed digital literacy was the most important
for students (Berardi, 2015).
As to teaching digital citizenship, only 57% of the respondents reported teaching a
lesson that corresponded to digital citizenship. Furthermore, only 58% of the teachers felt
they had enough self-efficacy to create new instructional experiences for students to
experience digital citizenship. Approximately half of the teachers, 51%, felt their
colleagues and supervisors supported or encouraged them to design digital citizenship
lessons. Only 33% of the respondents felt they were adequately supported. This was due
to a “lack of clarity regarding professional responsibility as it related to digital
citizenship” (Berardi, 2015, p. 57). Possible solutions to increase teacher efficacy for
digital citizenship instruction, as reported by respondents, included formal professional
development and providing a technology coach. Respondents particularly desired a
“clearer vision and mission from their district of employment to enhance their
selfefficacy regarding the instruction of digital citizenship” (Berardi, 2015, p. iv).
Challenges faced by the respondents included difficulties in
•identification of critical competencies,
•reacting to students’ digital violations,
•maintaining current knowledge amidst ongoing updates in technology,
•lack of an enhanced digital citizenship knowledge base with formal training •
lack of time for peer collaboration to develop best instructional practices (p.
63).
Suppo (2013) conducted a study of administrators to ascertain their perceptions of
digital citizenship and the frequency of digital citizenship instruction in their schools or
districts. While the administrators believed strongly that digital citizenship education was
important, “when comparing these same beliefs to the frequency digital citizenship
elements are taught, analysis showed a significant negative correlation” (Suppo, 2013, p.
88). Study findings suggested that “there may not be the strong relationship one would
assume between leadership beliefs and digital citizenship curriculum practices” (Suppo,
2013, p. 88).
The study revealed “the majority of administrators (83%) indicated that digital
citizenship is addressed within their districts while only 33% seemed to do so with
organized effort. Administrator responses indicated that 17% of school districts teach
digital citizenship infrequently or not at all” (Suppo, 2013, p. 86). “Nearly 50% of
participants indicating digital citizenship is addressed within their districts do so across
the curriculum without a formal digital citizenship curriculum” (Suppo, 2013, p. 89).
Based on the results of the study, Suppo recommended districts design a curriculum by
first conducting a digital citizenship audit to determine which resources are most needed.
The purpose of the current study, the creation of a walk-through document measuring the
demonstrable digital literacy proficiency of classroom teachers, could serve as a data
source for the digital citizenship curriculum audit Suppo recommends.
Districts were then, according to Suppo (2013), to design a digital citizenship
curriculum beginning in elementary school. The curriculum requires “hands-on authentic
learning activities [which] can be used within technology specific classes and/or woven
within existing classes where technology is being utilized to enhance learning
experiences” (Suppo, 2013, p. 95). Professional development for educators to increase
digital citizenship awareness and skills is necessary prior to implementing the curriculum.
Suppo also called for developing a tool to measure the level of digital citizenship
demonstrated daily by administrators, teachers, and students as further necessary research
on the topic of digital citizenship education.
Jones and Mitchell (2015) pointed out the ineffectiveness of digital citizenship
education as it is currently being taught in schools. Jones and Mitchell state, “the lack of
conceptual clarity of the term has hindered educational initiatives” (p. 14) as one reason
digital citizenship instruction is ineffective. There were a number of conflicting
definitions and models of digital citizenship as demonstrated by other authors noted in the
literature review. Jones and Mitchell (2015) broadened and refined the definition of
digital citizenship to include “using Internet resources to have youth practice respectful
and tolerant behaviors toward others and … increase civic engagement activities” (p. 3).
The pair developed a digital citizenship scale and conducted a study with a scale
involving 979 youth from ages 11-17. These students self-reported their online behavior
based on the digital citizenship definition above.
After examining the results of their study, Jones and Mitchell recommended
differentiating digital literacy education from digital citizenship education. “Digital
literacy requires a very specific set of educator knowledge and teaching skills compared
to other goals currently under the digital citizenship umbrella” (p. 2). The study also
concluded that typical digital citizenship education fails in one or more of these critical
components: “well-defined, includes active learning and other effective educational
strategies, target specific goals and outcomes, evaluate impact of intended behavioral
outcomes” (p. 12).
Crockett and Churches (2018) identified the differences between digital
citizenship and global digital citizenship. Global digital citizenship was defined as “how
we participate and contribute in blended physical and digital worlds” (Crockett &
Churches, 2018, p. 4). Tenets of global digital citizenship include personal responsibility,
global citizenship, digital citizenship, altruistic service, and environmental stewardship.
Crockett and Churches (2018) documented a number of criteria for schools to use
when crafting digital citizenship guidelines such as creating a student agreement outlining
respect and responsibility for the student, others, and property. A corresponding
agreement for teachers was also recommended.
Crockett and Churches (2018) created a digital citizenship analysis tool for
schools to use to evaluate digital citizenship instruction by examining the Acceptable Use
Policy, instruction, implementation, support resources, monitoring and consequences, and
community involvement. The authors also established a Global Digital Citizen
Foundation to expand on and share in creating global digital citizens.
Ribble, in 2006, completed a dissertation in which his goal was to research,
develop, test, and validate a handbook on digital citizenship for technology leaders.
During the course of his work, Ribble identified nine key themes of digital citizenship
education. He was also credited with coining the phrase and with the first definition of the
term. The working definition of digital citizenship for the purpose of the current study is
“the norms of behavior with regard to technology use” (Ribble, 2006, p. 1). Farmer
(2011) built on the work of Ribble and Bailey (2004) to expand the definition of digital
citizenship as “the ability to use technology safely, responsibly, critically, productively,
and civically” (p. 388). Additional researchers expanded on this initial definition and
proposed professional development for teachers to make them more proficient in teaching
digital citizenship to students. Table 5 contains a summary of the literature reviewed on
the topic of digital citizenship.
Table 5
Summary of Research on Digital Citizenship
Author Date Purpose
Ribble & Bailey 2004 Coined definition of digital citizenship and identified nine
traits
Ribble 2006 Refined nine traits; conducted study regarding a book about
digital citizenship for teachers
Farmer 2011 Outlined framework for adding informational practices to
digital citizenship education
Searson et al. 2015 Compared various models of digital citizenship education
Hollandsworth et
al.
2011 Surveyed school librarians regarding perceptions of digital
citizenship awareness in schools; found only half felt
teachers had suitable awareness
Berardi 2015 Studied elementary teachers to ascertain perceptions of
value and self-efficacy in digital citizenship; 96% believed
there was a need, yet only 58% had enough self-efficacy to
create lessons
Table 5 (continued)
Author Date Purpose
Suppo 2013 Surveyed administrators about perceptions of digital
citizenship and frequency of digital citizenship instruction;
found nearly 50% had no formal curriculum
Jones & Mitchell 2015 Broadened definition of digital citizenship; recommended
differentiating digital citizenship from digital literacy
Crockett &
Churches
2018 Described global digital citizenship; dementing criteria for
schools for implementing
Literacies
There are several literacies that correlate with digital citizenship such as digital
literacy, information literacy, internet literacy, media literacy, and web literacy. These
literacies are indistinct and overlapping. Just as researchers have been unable to identify
an exact definition of digital citizenship, there is also a lack of agreement on the aspects
of these various literacies.
Digital Literacy
What follows is a chronological review of the literature as the digital literacy field
has evolved. The phrase digital literacy refers to the problem-solving application of skills
practiced in digital citizenship education.
Ribble (2009) included digital literacy as one of the components of digital
citizenship education. It may also be said that digital literacy is a broader field that has
spurred research and definitions independent of digital citizenship instruction. Digital
literacy research pre-dated the initial Ribble work and may be considered the practical
application of digital citizenship skills.
Gilster (1997) first popularized the term digital literacy, defined as “the ability to
understand and use information in multiple formats from a wide range of sources when it
is presented via computers” (p. 1). The term also referred to the ability to “access
networked computer resources and use them” (Gilster, 1997, p. 1) and to solve problems
using search methods and an understanding of accessing information in a hypertext
format. Finally, digital literacy included “creating an information cache that access the
processing power of networked computers” (p. 33).
Gilster likened the necessity for developing digital literacy skills in the age of the
internet to the necessity for most adults to possess a driver’s license and the skills
requisite to drive a vehicle. The most essential digital literacy skill, according to Gilster,
was the ability to make an “informed judgement” of material found on the internet due to
the unfiltered nature and participatory nature of online content (p. 2). Other necessary
skills included being able to develop techniques to efficiently conduct online searches in
order to retrieve information and to assemble reliable information from diverse sources.
This was accomplished via developing critical thinking skills to reinforce network skills
in order to evaluate content and make intelligent decisions based on information found
online.
Digital literacy models. Ba, Tally, and Tsikalas (2002) refined the definition for
digital literacy as a “set of habits through which youngsters use information technologies
for learning, work, and fun” (p. 3). Digital literacy also encompassed skills such as
troubleshooting abilities, purposes for connecting, and skills in using popular technology
tools, communication literacy, and web literacy (Ba et al., 2002). The team recognized
creating definitions of digital literacy could be problematic due to rapidly changing
technologies that quickly render specific definitions obsolete. “The existing literature
uses various concepts to describe what we have defined as information technology
literacy, a hybrid framework encompassing skills with tools, computing concepts and
abilities, and cognitive capabilities toward solving real-world problems using technology”
(Ba et al., 2002, p. 10).
Ba et al. (2002) developed a digital literacy analysis model with five components:
computing for a range of purpose, understanding the function of and ability to use
common tools, communication literacy, web literacy, and troubleshooting skills. They
noted an assessment is particularly difficult to develop, in part due to a “lack of consensus
on what constitutes measurable dimensions of digital literacies” (Ba et al., 2002, p. 10).
Recommendations based on the study included developing assessment instruments to
measure digital literacy competence, training for teachers in how to support digital
literacy skill development in students, and conducting further research into instructional
techniques fostering the development of digital literacy (Ba et al., 2002).
Aviram and Eshet-Alkalai (2006) described digital literacy as a “survival skill in
the digital era” (p. 102). They noted research to date has been practice oriented and
lacked a theoretical foundation. They proposed a five-trait theoretical model for digital
literacy skills:
•Photo-Visual Literacy: Learning to read from visuals;
•Reproduction Literacy: The art of creative duplication;
•Branching literacy: Hypermedia and thinking or multiple-domain thinking;
•Information Literacy: The art of always questioning information;
•Socio-Emotional Literacy: The emotional and social aspects of working in
cyberspace.
The theoretical framework was then tested on 60 participants ranging from high
school aged students to adults in their mid-thirties. Participants were asked to complete
tasks in each of the five areas identified by the researchers.
•Photo-visual literacy: Decipher the graphic user interface and use a
multimedia program to construct a theatre stage.
•Reproduction literacy: Manipulate a given digital text in order to assign a
new meaning to it.
•Branching literacy: Design a tour to an unknown country through surfing the
Internet in a non-linear way.
•Information literacy: Write a critical comparison of the same piece of news
that was published in seven different Internet news sources.
•Socio-emotional literacy: Content analysis of inputs of participants in a chat
session (Aviram & Eshet-Alkalai, 2006).
The adolescents performed better on the technical aspects of the framework; older
participants performed better at the critical thinking and problem-solving aspects. Five
years later, 51 of the 60 original participants were re-tested along with a control group of
another 60 participants in the same age ranges. Paired t-tests compared the original
participants with their performance five years later and compared their performance to the
control groups.
Two major patterns emerged from the reassessment. All of the age groups
improved in technological skills and the younger participants actually performed worse at
the creative and problem-solving tasks. Based on this result, the researchers suggested
that “experience with technology, and not age, accounts for the observed lifelong changes
in digital literacy skills.” (Eshet-Alkalai & Chajut, 2009, p. 713).
The Eshet-Alkalai theoretical model was re-evaluated in 2012 and a sixth
component, real time thinking, was added. Real time thinking represented “the ability to
process large volumes of stimuli at the same time, as in video games or in online
teaching” (Eshet, 2012, p. 267).
Blummer (2008) discussed the use of digital literacy competencies and their
importance to students as educators “struggle to engage students as well as provide
instruction relevant to their future academic and work environments” (p. 38).The working
definition for digital literacy to Blummer included “a range of abilities from basic
computing skills to the creation of multimodal texts” (p. 38), which was essential as
students applied the digital practices in their personal lives to their academic
environment. The addition of creation aspects to the latest iteration of digital literacy
demonstrated a large shift from the original definition in 1997 that mentioned merely the
use of digital tools.
Blummer (2008) stated there was a need for more study on digital literacy.
Blummer noted “the topic of digital literacy is characterized by a nebulous definition and
limited research studies that are often centered in foreign countries” (p. 44). Nevertheless,
it was “essential to correlate student digital literacy habits in their personal lives to
instructional practices at school” (Blummer, 2008, p. 38).
Hobbs (2010) defined digital literacy as “a constellation of life skills that are
necessary for full participation in our media-saturated, information-rich society” (p. vii).
These included the ability to do the following:
•Make responsible choices and access information by locating and sharing
materials and comprehending information and ideas.
•Analyze messages in a variety of forms by identifying the author, purpose,
and point of view, and evaluating the quality and credibility of the content.
•Create content in a variety of forms, making use of language, images, sound,
and new digital tools and technologies.
•Reflect on one’s own conduct and communication behavior by applying social
responsibility and ethical principles.
•Take social action by working individually and collaboratively to share
knowledge and solve problems in the family, workplace, and community and
by participating as a member of a community. (p. viii)
Hobbs (2010) identified several related terms—such as information literacy,
media literacy, media education, visual literacy, news literacy, and digital literacy—and
how they were related.
For example, information literacy has typically been associated with research
skills. Media literacy typically has been associated with critical analysis of news,
advertising and mass media entertainment… Digital literacy is associated with the
ability to use computers, social media, and the Internet. Although they reflect
distinct and important theoretical ideas and values from different disciplinary
traditions and historical contexts, effective programs in all of the “new media
literacies” reveal many similarities. (p. 17)
Hobbs (2010) proposed using the term digital and media literacy to greater
encompass the complementing skills sets required to interpret texts, tools, and techniques
students encounter. Digital media literacy was necessary for students to not only
strengthen “capacity for engaging with information but also for addressing potential risks
associated with mass media and digital media” (Hobbs, 2010, p. 29).
The report also made a number of recommendations for educators to consider
when creating a plan for digital literacy instruction:
•moving beyond a tool-oriented focus that conflates having access to media
and technology with the skillful use of it;
•addressing risks associated with media and digital technology;
•expanding the concept of literacy;
•strengthening people’s capacity to assess message credibility and quality; and
•using news and journalism in the context of K–12 education (p. xii).
In order to effectively accomplish these recommendations, Hobbs (2010) called
for teachers to have an opportunity to work with and observe peers, a collection of
teacher-created, peer-reviewed lesson plans, and further research into best practices for
teaching digital and media literacy. Developing an assessment of digital and media
literacy competence for teachers was also suggested.
The American Library Association (2013) highlighted the necessity for digital
literacy in order to function in a global, connected society in the report from the Office
for Information Technology Policy’s Digital Literacy Task Force. The American Library
Association Digital Literacy Task Force reported 80% of Fortune 500 companies posted
open positions online and required online applications and the majority of modern era
jobs required some level of technology skills. The organization defined digital literacy as
“the ability to use information and communication technologies to find, understand,
evaluate, create and communicate digital information” (American Library Association,
2013, p. 1). A digitally literate person:
•Possesses the variety of skills—cognitive and technical—required to find,
understand, evaluate, create, and communicate digital information in a wide
variety of formats;
•Is able to use diverse technologies appropriately and effectively to search for
and retrieve information, interpret search results, and judge the quality of the
information retrieved;
•Understands the relationships among technology, lifelong learning, personal
privacy, and appropriate stewardship of information;
•Uses these skills and the appropriate technologies to communicate and
collaborate with peers, colleagues, family, and on occasion the general public;
•Uses these skills to participate actively in civic society and contribute to a
vibrant, informed, and engaged community. (p. 2)
Digital literacy studies. Li and Ranieri (2010) conducted a study that utilized the
Instant Digital Competence Assessment developed by Calvini et al. (2008) on a group of
317 Chinese ninth grade students. Although they lived in the same city, there was a wide
variation of self-reported digital competence among members of the group. Despite the
range of self-reported technical skills, the average performance level students
demonstrated was “pass” rather than “good” or “excellent.”
Li and Ranieri asserted as a result of their study, “[a]lthough accessing the Internet
is not a problem for the majority of them, students might lack important skills, such as
inquiry skills and critical thinking skills, to correctly analyze and efficiently use those
online resources and digital tools” (p. 1039). The study concluded that frequent use of
technology does not necessarily result in greater digital competence.
Simsek and Simsek (2013) made the connection between digital and other
emerging literacies as they applied to student use in their daily lives. Digital literacies
emerged primarily “because traditional literacy could not meet the requirements of the
digital age, more complicated skills are needed in the name of new literacies. In the
digital age, it is a vital requisite to fully understand and use the capacity of new
information and communication technologies” (Simsek & Simsek, 2013, p. 128).
Simsek and Simsek (2013) asserted these new literacies were “prerequisites for
digital citizenship. New literacies increased the availability of relevant and credible
information and broaden the capacity of individuals to get, share, compare, and
contextualize information by developing new skills” (Simsek & Simsek, 2013, p. 133).
However, students could be competent in the literacies without exhibiting literacy skills
for responsible citizenship. Simsek and Simsek implied through their statement that
digital citizenship instruction does not necessarily translate into practice.
Digital literacy, in addition to being one of the nine components of digital
citizenship, is also a freestanding collection of skills and competencies. The definition of
digital literacy “includes a range of abilities from basic computing skills to the creation of
multimodal texts” (Blummer, 2008, p. 38). Gilster (1997) first popularized the term
digital literacy, defined as “the ability to understand and use information in multiple
formats from a wide range of sources when it is presented via computers” (p. 1). Aviram
and Eshet-Alkalai (2006) described digital literacy as a “survival skill in the digital era”
(p. 102). Despite the need for digital literacy skills in order to fully participate in modern
networked society, research has demonstrated that teachers need more training in how to
effectively instruct students in digital literacy.
Information Literacy
Information literacy was originally coined in the 1974 by describing people who
work regularly with information, such as librarians, as being “information literate” by
Zurkowski, the president at the time of the Information Industry Association (p. 6).
Information literate individuals “have learned technique and skills for utilizing the wide
range of information tools as well as primary sources in molding information solutions to
their problems” (Zurkowski, 1974, p. 6).
The definition of information literacy included an understanding of how to “locate
and use information needed for problem-solving and decision-making efficiently and
effectively” (Behrens, 1994, p. 310). The definition included skills to locate and use
information, to use information for problem solving and decision making, and to
efficiently and effectively locate and utilize information. The definition of information
literacy in 1974 was later expanded to include the following elements: applying facts and
information to problem-solving, understanding that a variety of information resources are
available, recognizing that a continual information process exists, and applying strategies
to acquire information (Behrens, 1994). Specific information literacy characteristics were
outlined in 1985. These characteristics included “an integrated set of skills and
knowledge… developed through acquisition of attitudes” that are time and labor
intensive, need-driven, and “distinct but relevant to literacy and computer literacy”
(Behrens, 1994, p. 312).
Koltay (2011) described information literacy, which predated digital literacy, as
“the process of recognizing information need, finding, evaluating and using information
to acquire or extend knowledge” (p. 33). Information literacy education prioritized
critical thinking, especially in an age where “gatekeepers” such as editors and reviewers
were not present online the same way they were in printed materials. This necessitated
that the consumer of information also became the gatekeeper of said information.
Miller and Bartlett (2012) made the connection between digital literacy and
information literacy in that digital literacy combines “old techniques” such as critical
engagement with information “with new and specific knowledge bases about how the
internet works, and how, given how it works, it can inadvertently deceive or be
deliberately used to deceive” (p. 39).
Lanning (2014) defined information literacy instruction as “a planned instruction
session for multiple recipients in a formal setting with the goal of imparting IL
(information literacy) skills” (p. 79). One challenge identified was timeliness in that
information literacy instruction may not be as effective if it is not done at a time when
students need to interact with information in a particular way. Planning information
literacy instruction to correspond with curricular needs or student projects was deemed
more impactful than stand-alone lessons.
Lanning (2014) recommended an information literacy program be implemented
on a campus based on curricular information needs and not on a specific research process.
A blended learning environment, one that utilizes both face-to-face and online instruction
was one method for implementing such a program.
Alvarez and Gisbert (2015) defined information literacy as “the ability to treat
information and to use this information to construct knowledge and lifelong learning in
order to solve any problems we may encounter” (p. 188). The pair conducted a study of
2,656 secondary school teachers in Spain to determine their self-perceived level of
information literacy utilizing the Secondary Education Schoolteacher Information
Literacy Questionnaire developed for the study. The questionnaire was based on the
Digital Competence Assessment created by Calvini et al. (2008) for use with students.
The Likert-scale questionnaire assessed teachers on several indicators—recognizing the
need for information, locating, evaluating, organizing and transforming information. Not
only were the teachers questioned about their self-efficacy with information literacy, they
were also asked to demonstrate proficiency using simulations and practical applications.
Participant responses determined 87.8% of the teachers felt comfortable searching
information on the internet (p. 190), yet only 26% reported that they modified the
information before using it or identified the author of the information found (p. 192). The
authors found in general that teachers had a higher self-perception of their level of digital
literacy than they were able to demonstrate, particularly in the indicators of evaluating
and organizing information.
Alvarez and Gisbert (2015) noted participants “present major deficiencies when it
comes to evaluating the information they find” (p. 192). Teachers also demonstrated
difficulty determining the credibility of information on the internet “as reliable and true in
comparison with information they obtain from analogical sources (only 52.4% do)
(Alvarez & Gisbert, 2015, p. 192).
Recommendations based on the study included teacher training “both in
producing and disseminating information and in evaluating and managing information.”
The training “should focus on the specific aspects and indicators we have mentioned in
this study regarding the evaluation, organization, management and transformation of
information” (Alvarez & Gisbert, 2015, p. 193).
The definition of information literacy included an understanding of how to “locate
and use information needed for problem-solving and decision-making efficiently and
effectively” (Behrens, 1994, p. 310). Information literacy has also been defined as “the
process of recognizing [an] information need; finding, evaluating and using information
to acquire or extend knowledge” (p. 33). Research has demonstrated both students and
teachers can benefit from increased attention to developing information literacy skills
(Alvarez & Gisbert, 2015, Stanford History Education Group, 2016).
Internet and Web Literacy
Johnson (2007) discussed various internet literacy definitions as “the capability to
access and evaluate online information,” online search competence, and “skill with
connectivity, security, communication, multimedia, and web page development” (p. 434)
in an article outlining the required cognitive skills for functional internet literacy. These
cognitive skills included higher-order thinking skills such as the ability to analyze,
synthesize, and evaluate online activities. Users must be able to discern factual from false
information and determine legitimate posters of information online. “Functional Internet
literacy, which requires complex cognitive processing, is best achieved in structured and
directed learning situations. Formal teaching of functional Internet literacy is based upon
the assessment of cognitive skill deficits and instruction that targets those identified
deficits” (Johnson, 2007, p. 438). A summary of cognitive skills required for common
internet use, as identified by Johnson (2007), is included in Table 6.
Table 6
Required Cognitive Skills for Common Internet Use
Internet Use Cognitive Skill Demonstration of Cognitive Skill
Communication Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Enter email address
Read and understand message
Comply with email instructions
Identify essential information
Combine multiple messages
Judge intention of communicator
Information Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Enter search term
Understand site navigation
Use information in daily life
Determine site author
Summarize site information
Evaluate site information
Recreation
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Enter game site URL
Understand game rules
Locate similar games online
Determine game elements
Generalize skill from other games
Judge quality of game
Commercial Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Enter credit card information
Understand product information
Compare with real products
Identify critical components
Combine multi-site information
Decide on best product
Technical Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Launch browser
Explain importance of browser
Use alternative browser
Compare browser features
Align browser with requirements
Recommend browser
Note: Adapted from “Functional Internet Literacy: Required Cognitive Skills with
Implications for Instruction,” by Genevieve Marie Johnson, 2007, E-Learning and
Digital Media, 4(4), p. 437.
November (2012) is the prominent voice of web literacy as a necessary
component of digital literacy. November defined web literacy as “accurate and creative
searching techniques that are applicable across every discipline” (p. 50). “Web literacy
isn’t just about learning to be productive online; it also involves taking control of the
technologies that we use, so that we—not our tools—are guiding our results” (p. 72).
A lack of web literacy skills is problematic because students are “using paper
literacy skills to navigate in a digital world and coming up with misleading and shallow
results” (November, 2012, p. 50). Students are often “web-illiterate” despite daily use of
technology and the Internet. Students must be educated, according to November, on how
to assess the origin of information and how to validate that information. Students must
also learn how to “research, publish, and communicate through and with the Internet and
other information tools” (p. 52).
Miller and Bartlett (2012) also referred to the lack of a gatekeeper on the internet,
similar to what Koltay (2011) did, in regard to information literacy. User-generated
content on the internet necessitated the end user to develop internet literacy skills.
According to Miller and Bartlet (2012), the following situations illustrate the necessity
for internet literacy:
•Pseudo-sites and propaganda: websites created for nefarious purposes and are
specifically designed to appear trustworthy
•Use of imagery: Image manipulation techniques allowing misinformation to
be powerfully and attractively packaged
•Echo chambers: Information mediated by algorithms to create a personalized
online experience populated by essentially agreeable information
•“Skittering” and “bouncing”: Visiting only one to three pages from the
thousands that are available (bouncing), and viewing more than reading
(skittering). (p. 37)
To summarize the various literacies described in the literature review, the working
definition of digital citizenship for the purpose of the current study is “the norms of
behavior with regard to technology use” (Ribble, 2006, p. 1). Digital citizenship has
alternately been defined as “appropriate technology usage” and “making safe,
responsible, respectful choices online” (Common Sense Media, 2018). Digital literacy, in
addition to being one of the nine components of digital citizenship, is also a freestanding
collection of skills and competencies. The definition of digital literacy “includes a range
of abilities from basic computing skills to the creation of multimodal texts” (Blummer,
2008, p. 38).
Information literacy is “the process of recognizing information need, finding,
evaluating and using information to acquire or extend knowledge” (Behrens, 1994, p. 33).
Internet literacy is “the capability to access and evaluate online information,” online
search competence, and “skill with connectivity, security, communication, multimedia,
and web page development” (Johnson, 2007, p. 434). These literacies are all interwoven
and required skills overlap. Research has demonstrated they all are crucial to students as
they learn to navigate a global, connected world. Table 7 contains a summary of the
literature reviewed on the various literacies discussed.
Technology Integration
Teachers can experience difficulties in teaching digital citizenship, digital literacy,
and/or web literacy skills to students as they are often learning the content along with
students and may be less familiar with the technology than the students they educate. This
was noted by several researchers listed above. Ribble classified the dilemma as a “double
learning curve” (2006, p. 118). For teachers, “the challenge of keeping up with students
as they create and publish in ever-increasing numbers are daunting” (Richardson, 2009, p.
26).
Table 7
Summary of Research on Literacies
Topic Author Date Purpose
Digital Literacy Gilster 1997 Invented the term and definition for digital
literacy
Digital Literacy Ba et al. 2002 Developed digital literacy analysis model
with five components
Digital Literacy Aviram & Eshet-
Alkalai
2006, 2012 Developed and tested a framework for digital
literacy
Digital Literacy Blummer 2008 Noted a nebulous definition and a need for
more research in the United States
Digital & Media
Literacy
Hobbs 2010 Widened definition of digital literacy to
include information literacy and media
literacy; called for an assessment of teacher
competency
Digital Literacy Li & Ranieri 2010 Conducted study on 317 students; found
access to technology did not improve
digital literacy skills
Digital Literacy Simsek & Simsek 2013 Expanded digital citizenship to include
digital and emerging literacies; asserted
digital citizenship instruction did not
necessarily translate into practice
Information
Literacy
Zurkowski 1974 Defined the term
Information
Literacy
Koltay 2011 Determined the consumer of information also
acts as gatekeeper of information in the
digital age
Information
Literacy/ Internet
literacy
Miller & Bartlett 2012 Made connection between information and
digital literacies; discussed necessity for
internet literacy
Information
Literacy
Lanning 2014 Outlined weaknesses of traditional
information literacy instruction and made
recommendations for more effective
instructional methods
Table 7 (continued)
Topic Author Date Purpose
Information
Literacy
Alvarez & Gisbert 2015 Conducted a study on teachers to determine
how they implemented information literacy
practices in the classroom; found “major
deficiencies” in evaluating information
Internet & Web
Literacy
Johnson 2007 Identified required skills for functional
internet literacy
Internet & Web
Literacy
November 2012 Made recommendations to teachers regarding
strategies to increasing internet literacy skills
practice for students
Farmer (2011) provided a partial roadmap for how to evaluate digital information
by engaging students in “active examination, debate, and self-reflection, educators can
use a variety of technological tools: threaded discussion, online chat, blogs, wikis, and
online conferencing” (p. 390), yet the task still proves daunting to many educators,
especially those first encountering the Internet as adult learners.
November (2012) encouraged the role of the teacher to evolve into that of a global
publisher of student work and to redefine the role of the student as a contributor in a
“collaborative learning culture” (p. 45). November advocated the role of the teacher
continually evolve as new information sources become available.
Teaching in the redefined role must include “techniques for applying knowledge
to produce information and facilitate communication…evaluating the resources they
decide to use” (November, 2012, p. 52). Likewise, students must be taught to examine all
information found on the Internet for purpose, author, and place and how the information
found on a particular website is linked to other information found on other websites.
Models of Technology Integration
The two most common models of technology integration, with the exception of
the ISTE standards, are described below. They are the SAMR (Substitution,
Augmentation, Modification, Redefinition) model and the TPACK (Technological
Pedagogical Content Knowledge) model.
SAMR model. The SAMR model was developed by Dr. Ruben R. Puentedura in
2006 working with the Maine Department of Education on the Maine Learning
Technology Initiative (Puentedura, 2006). The framework is a continuum or progression
composed of four stages divided into two levels of technology integration, as
demonstrated in Figure 1.
Figure 1. SAMR Model, Puentedura (2006).
The lowest stage of technology integration is Substitution, where “technology acts
as a direct tool substitute with no functional change” (Puentedura, 2006, p. 2). An
example of an instructional practices at this level would be to use technology to complete
a worksheet that had been digitized. The second stage of technology integration is
Augmentation in which technology “acts as a direct tool substitute with functional
improvement” (Puentedura, 2006, p. 2). An example of an instructional practice at the
substitution level would be to take notes in a digital format as opposed to writing them
into a notebook. These two stages incorporate the Enhancement level of technology
integration.
The third stage of the SAMR model is Modification in which technology “allows
for significant task redesign” (Puentedura, 2006, p. 2). An example of a redefined
assignment would include one with online collaboration. The final, and highest, stage of
technology integration is Redefinition. In the Redefinition stage, technology “allows for
the creation of new tasks, previously inconceivable” without technology (Puentedura,
2006, p. 2). An example of an instructional practice that would be impossible without
technology would include student-created videos or student-built websites. These last two
stages are the Transformation level in which learning is transformed by technology.
Studies about SAMR model. Patton (2015) conducted a phenomenological
narrative study to explore how teachers implemented 1:1 technology integration in the
classroom based on the SAMR model as the theoretical framework. The study explored
challenges that were encountered during the integration of technology into the classroom
and professional development required for successful technology integration at the
various levels of the SAMR model.
Patton (2015) interviewed seven math and science teachers to explore how 1:1
technology was integrated into their classrooms based on the SAMR model. Patton
(2015) found that despite receiving at least one professional development session on the
SAMR model, most participants were unable to give examples of technology-driven
instructional activities they had implemented in their classrooms at the Modification and
Redefinition level (p. 86). In addition, the majority of instructional activities involving
technology occurred at the Enhancement level, or the Substitution and Augmentation
stages, of technology integration. “Out of the technology-driven instructional activities
that were observed 72% were at substitution, 14% were at augmentation, 4% were at
modification, and 10% were at the redefinition levels” (Patton, 2015, p. 86). One reason
for this as reported by Patton (2015) was that while teachers had a basic knowledge of the
SAMR model levels, they did not know how to effectively integrate the model into
instruction. Patton (2015) recommended more professional development as a result.
Pfaffe (2017) conducted a study evaluating mobile learning (mLearning) tools and
applications used by 103 secondary teachers using the SAMR framework. Teachers from
23 states and Mexico completed an online survey to self-report how mobile learning was
integrated into their instruction.
The Pfaffe (2017) study findings were similar to those of Patton (2015). Findings
indicated most learning activities reported by participants occurred at the Enhancement
levels (Substitution and Augmentation) of the SAMR model. Again, this was due to a lack
of professional development for teachers in how to implement technology at the
Transformation level. In addition, only one district involved in the study used a
technology integration framework in their district professional development.
Recommendations from Pfaffe (2017) included additional professional development in
technology integration at the Transformation level of the SAMR model.
“Professional development should also support faculty and administrators in technology
visioning, planning, and building capacity and investigate new technology standards
established by ISTE and the Partnership for 21st Century Skills” (Pfaffe, 2017, p. 165).
TPACK framework. The TPCK, or TPACK, framework originated in 1986 with
Shulman, who proposed Pedagogical Content Knowledge, or PCK, as a form of
knowledge teachers needed to possess in addition to knowledge of subject matter and
curricular knowledges. PCK included “an understanding of what makes the learning of
specific topics easy or difficult: the conceptions and preconceptions students of different
ages and backgrounds bring with them to the learning of those most frequently taught
topics and lessons” (Shulman, 1986, p. 9).
In 2006, Mishra and Koehler added a technology component in an effort to further
teacher integration of technology into pedagogy. “[T]houghtful pedagogical uses of
technology require the development of a complex, situated form of knowledge that we
call Technological Pedagogical Content Knowledge (TPCK). In doing so, we posit the
complex roles of, and interplay among, three main components of learning environments:
content, pedagogy, and technology” (Mishra & Koehler, 2006, p. 1017). The TPCK or
TPACK model is demonstrated in Figure 2.
•“Content knowledge (CK) is knowledge about the actual subject matter that is
to be learned or taught” (Mishra & Koehler, 2006, p. 1026).
•“Pedagogical knowledge (PK) is deep knowledge about the processes and
practices or methods of teaching and learning and how it encompasses, among
other things, overall educational purposes, values, and aims” (Mishra &
Koehler, 2006, p. 1026).
•The idea of pedagogical content knowledge (PCK) is consistent with, and
similar to, Shulman’s idea of knowledge of pedagogy that is applicable to the
teaching of specific content. This knowledge includes knowing what teaching
approaches fit the content, and likewise, knowing how elements of the content
can be arranged for better teaching. (Mishra & Koehler, 2006, p. 1027)
•“Technology knowledge (TK) is knowledge about standard technologies, such
as books, chalk and blackboard, and more advanced technologies, such as the
Internet and digital video” (Mishra & Koehler, 2006, p. 1027).
•Technological content knowledge (TCK) is knowledge about the manner in
which technology and content are reciprocally related. Although technology
constrains the kinds of representations possible, newer technologies often
afford newer and more varied representations and greater flexibility in
navigating across these representations. (Mishra & Koehler, 2006, p. 1028)
•Technological pedagogical knowledge (TPK) is knowledge of the existence,
components, and capabilities of various technologies as they are used in
teaching and learning settings, and conversely, knowing how teaching might
change as the result of using particular technologies. This might include an
understanding that a range of tools exists for a particular task, the ability to
choose a tool based on its fitness, strategies for using the tool’s affordances,
and knowledge of pedagogical strategies and the ability to apply those
strategies for use of technologies. (Mishra & Koehler, 2006, p. 1028)
Figure 2. TPACK model, Mishra & Koehler (2006).
TPACK holistically combines all of the above elements. The teacher uses
technology in relation to pedagogy, understands how to integrate technology into content,
and realizes how technology can be applied to build upon student knowledge of a
concept.
Quality teaching requires developing a nuanced understanding of the complex
relationships between technology, content, and pedagogy, and using this
understanding to develop appropriate, context-specific strategies and
representations. Productive technology integration in teaching needs to consider
all three issues not in isolation, but rather within the complex relationships in the
system defined by the three key elements. (Mishra & Koehler, 2006, p. 1029)
Studies about TPACK framework. Green (2017) conducted a study comparing
teacher self-identified TPACK proficiency with student achievement. A total of
252 educators completed a 50-question survey regarding their level of TPACK
and were classified in one of three categories—low, medium, and high. The high-
TPACK group, according to demographic data, tended to be younger teachers and
more likely to be male. Teachers rated at a mid-TPACK level contained mostly
older and female teachers. The low-TPACK group consisted primarily of
educators toward the end of their careers.
A chi-square test “was used to determine whether teachers with higher or lower
levels of student performance are significantly more prevalent in some of the TPACK
clusters” (Green, 2017, p. iii). The chi-square test found students with teachers in the
high-TPACK group demonstrated a significantly higher level of achievement with a score
of χ2(8) = 15.851, p = .045. (Green, 2017, p. iii). A t-test showed students in the high-
TPACK group performed “significantly higher” than those in the mid-TPACK group t(97)
= -3.045, p = .004. The t-test showed an average difference of 18.2579 among student
scores between the two groups of students (Green, 2017, p. iv). Based on these results,
Green determined teachers with a high level of TPACK proficiency produced a higher
level of student achievement than those with a middle or low level of technology
integration expertise.
McCusker (2017) examined the efficacy of TPACK professional development on
technology integration of teachers. The mixed method study consisted of an online survey
of 64 participants and interviews with 17 of the participants. Of the various approaches to
TPACK professional development, the PCK methods were offered the most often to
participants. PCK methods included “structured instructional modeling, peer coaching,
and/or collaborative development of instructional materials situated within particular
curriculum areas” (McCusker, 2017, p. 105). Participants noted they often practiced
informal peer coaching among themselves or participated in curriculum writing.
Participants also noted a lack of “fit” in the professional development they
attended, which was problematic because effective context is crucial to TPACK.
McCusker (2017) also found limited campus-based, job-embedded support after the
professional development hindered teacher progress in implementing what was learned in
the sessions. In addition to adding this support, McCusker recommended providing
professional development on the Reflective/Reflexive Methods and Computer-Adaptive
Methods aspects of TPACK technology integration, which were missing from the
professional development offered to participants during the study.
Two of the most common technology integration models were described
previously. Research on the SAMR model demonstrated teachers mainly focused
instruction on Enhancement, the lower end of the spectrum (Patton, 2015; Pfeffe, 2017).
The researchers concluded teachers needed additional support and training in order to
more effectively integrate technology at the higher, transformative level.
The TPCK, or TPACK, framework originated in 1986 with Shulman, who
proposed PCK as a form of knowledge teachers needed to possess in addition to
knowledge of subject matter and curricular knowledge. It was broadened in 2006 by
Mishra and Koehler to include technology integration. Research on the TPACK model
demonstrated current technology- and curriculum-related professional development
contained gaps in coverage regarding the TPACK model (McCusker, 2017). However,
teachers with a high proficiency in TPACK produced a higher level of student
achievement (Green, 2017), so professional development was found to be a necessary
component for teachers not operating at a high level of technology integration. Table 8
summarizes the literature reviewed regarding technology integration.
Table 8
Summary of Research on Technology Integration
Author Date Purpose
Puentedura 2006 Created SAMR (Substitution, Augmentation,
Modification, Redefinition) model
Patton 2015 Studied teacher 1:1 technology integration using SAMR
framework; found teachers primarily still use lower
levels in instruction
Pfaffe 2017 Studied mobile learning instructional practices based on
SAMR model; found teachers lacked professional
development to move beyond lower levels
Shulman 1986 Created PCK (Pedagogy Content Knowledge)
framework for instruction
Mishra &
Koehler
2006 Added technology component to create TPCK
framework for instruction
Green 2017 Compared teacher self-reported TPACK levels to
standardized assessment scores; found students
performed better in classrooms with teachers reporting
a high level of TPACK proficiency
McCusker 2017 Studied professional development through TPACK lens;
identified gaps in professional development that might
hinder TPACK implementation
Walk-through
A walk-through was defined by The Center for Comprehensive School Reform
and Improvement (2007) as a “brief, structured, nonevaluative classroom observation by
the principal that is followed by a conversation between the principal and the teacher
about what was observed” (p. 1). A walk-through occurs in a shorter time span than that
of a formal observation, yet because it is performed more often, it allows an administrator
to see more classroom instruction over time than a one-time formal observation.
Frequent, consistent walk-throughs enable administrators to observe patterns of
instruction. The primary purpose of a walk-through is “to provide a structure for dialogue
between principal and teacher about what goes on in the classroom” (The Center for
Comprehensive School Reform and Improvement, 2007, p. 2).
Aspects of Effective Walk-throughs
The Center for Comprehensive School Reform and Improvement (2007) outlined
several essential elements of a walk-through. They include brevity, focus, and dialogue.
Walk-throughs must be brief in order to allow the administrator to maximize the number
of classrooms in a given time period. Teachers should be given a purpose for the
walkthrough in order for it to be most effective. Lastly, an effective walk-through is a tool
to facilitate feedback and reflection among the teacher and administrator.
Sketta (2007) advocates walk-throughs as a professional development tool in
addition to the curricular focus described by The Center for Comprehensive School
Reform and Improvement. Prior to using walk-through evidence to guiding professional
development decisions, teachers must be familiar with the criteria in the walk-through
checklist.
Sketta (2007) suggested conducting professional development on the walkthrough
document itself prior to implementation. Also recommended was the practice of
identifying teachers demonstrating strengths in focus areas during the walk-through
process and utilizing those teachers as a professional development resource for other staff
members.
Cervone and Martinez-Miller (2007) described a walk-through as a method a
learning community can employ to “deepen its collective understanding of instruction—
moving beyond identifying and ‘fixing’ problems to identifying and enhancing student
mastery of content and skills” (p. 2). Walk-through results can also be used to measure
improvement or progress for an instructional program. Steps identified for implementing
walk-throughs included:
•Gather data and evidence of student engagement.
•Hypothesize on the behavior or outcome to change.
•Implement.
•Reflect on implementation.
•Repeat implement and reflect as necessary.
Gillespie (2016) described walk-throughs as a method to produce data for
“meaningful professional conversations” in order to promote instructional improvement
(p. 1). Purposes of walk-throughs included nurturing “collaboration, self-efficacy, and a
positive work environment that supports teachers in their professional growth (p. 1).
Gillespie (2016) identified several look-fors during a walk-through including
“climate for learning, clear learning objectives, cognitive level, student engagement,
teaching strategies, assessment techniques, and technology integration” (p. 7).
Walkthroughs also should have the following attributes:
•brevity,
•routine,
•unannounced and unobtrusive,
•structure and focus,
•black and white, and
•supportive.
Gillespie (2016) noted walk-throughs require a well-designed rubric to facilitate
brevity and efficiency. Walk-throughs are to be a regularly occurring habit for teachers,
students, and administrators in order to produce the most valid data.
Walk-through expectations should be clear and objective, which also require a
well-designed rubric with observable look-fors. The purpose of walk-throughs should be
affecting positive instructional change and not punitive in nature. A culture supportive of
walk-throughs is also a critical factor to success (Gillespie, 2016).
Successful walk-throughs also had several features regarding data keeping that
must be maintained (Gillespie, 2016,). These features include immediate feedback,
powerful reports, and data archiving in addition to the self-reflection identified by
Cervone and Martinez-Miller (2007).
Cooper (2015) recommended several components of a successful walk-through
document. The document should be housed digitally for immediate feedback and the
document, according to Cooper (2015) is to contain the following components:
•objective summary;
•specific feedback not limited to a checklist; preferably a question;
•targeted feedback with “I can” statements or bullet points;
•optional area for teacher reflection or an action plan; and
•multimedia such as photographs, video, or artifacts from the lesson.
Graf and Werlinich (2004) discussed the contributions walk-throughs make to
improving a school culture by increasing collaboration and a focus on students.
Administrators are to “observe students’ behaviors, level of engagement, and quality of
work” (p. 4). Graf and Werlinich (2004) also recommend administrators also discreetly
speak to students about what they are doing.
Rossi (2007) conducted a study on elementary school principals at campuses
utilizing the Walk-through Observation Tool developed by Graf and Werlinich (2004) to
evaluate how walk-throughs improve student learning. Perceptions of the principals
showed consensus on aspects of the model such as focus on the positive, establish
lookfors, be consistent, build trust, and establish expectations and guidelines (Rossi,
2007).
Rossi (2007) found that the walk-through process impacted principals as
instructional leaders and impacted instruction. Rossi (2007) also identified several
recommendations for administrators as they implement walk-throughs including to
validate effective practices and to collaboratively identify look-fors tied to school mission
and vision.
Protheroe (2009) advised a walk-through duration be between five and 15 minutes
and have a distinct purpose, or look-fors in order to be most effective. Protheroe (2009)
also highlighted benefits to walk-through observations when used to improve instruction
and aspects of the walk-through process requiring focus in order to maximize
effectiveness. Benefits of walk-throughs include:
•administrators become more familiar with school climate, curriculum and
instructional practices;
•team building culture can result as teachers and administrators work together
to improve instructional practices;
•administrators demonstrate instructional leadership via mentoring and
providing feedback; and students see administrators and teachers value
instruction and learning (p. 30).
Downey Walk Through Process
Downey, Steffy, English, Frase, and Poston (2004) outlined the Downey
WalkThrough Process in The Three-Minute Classroom Walk-Through. Goals of a
walkthrough including reflective dialogue were
•reflective, self-directed, self-analytical, interdependent teachers who examine
their own practices;
•teachers who are continually willing to improve their teaching practices; and
•teachers who are committed to teaching the district curriculum, student
learning, and working for ever higher student achievement (p. 8).
The Downey Walk-Through Process began in the 1960s as Downey first initiated
making classroom visits (Downey et al., 2004). It evolved over the years to include a
portion of the Madeline Hunter teacher evaluation approach, cognitive coaching,
reflective dialogue, transactional analysis, and Management by Wandering Around. The
Downey walk-through model was unique in its focus on “teacher decisions rather than on
teacher actions” (Downey et al., 2004, p. 12). The Downey Walk-Through contained five
ideas:
1. short, focused, informal observation;
2. possible reflection;
3. curriculum and instructional focus;
4. occasional follow-up; and
5. informal and collaborative.
Downey et al. (2004) highlighted a number of reasons walk-throughs were
important to instructional improvement. Reasons to conduct walk-throughs included the
following:
•frequent sampling leads to validity of observations;
•frequent observations are more effective;
•common areas can be identified for staff development;
•staff development implementation can be observed; and
•data can be used to identify teachers needing assistance (p. 6).
Prior to initiating a walk-through program, a campus administrator must prepare
students, staff, and parents. Preparation includes a teacher orientation outlining the
rationale for walk-throughs and steps of the process. A memo for teachers and parental
notification is also recommended.
The Downey Walk-Through is structured by five steps. These steps include:
1. student orientation to the work,
2. curricular decision points,
3. instructional decision points,
4. “walk-the-walls” curricular and instructional decisions, and
5. safety and health issues (p. 21).
Administrators are advised to take brief notes on a note card regarding the five
steps listed above. During the Downey Walk-Through Process, checklists are not to be
used.
Following a walk-through, Downey et al. (2004) recommend occasionally
engaging the teacher in a reflective conversation. This conversation is to last
approximately three to five minutes and begin with a positive statement about what was
observed during the classroom visit. The administrator then asks an unfocused question
about what occurred in the classroom to invite reflection and “probe for the criteria the
teacher uses in making a particular decision when one is brought up” (Downey et al.,
2004, p. 58). The administrator then summarizes the conversation and issues an invitation
for follow-up.
Administrators are advised to avoid certain practices in order to successfully
implement the model. These practices include assuming the model is merely a technical
approach, being unaware that one’s perspective is the “Normative Gaze” and being
unable to see change as a process and not an outcome (Downey et al., 2004, p. 166).
Other barriers to successful implementation include lack of teacher buy-in, lack of
adequate curriculum, a culture of teacher isolation, and a “clash of evaluative narratives”
in which district or state evaluation systems were resented by teachers as they grew
accustomed to walk-throughs (Downey et al., 2004, p. 173).
Comparison of Other Walk-through Models
Kachur, Stout, and Edwards (2010) conducted an examination of 18 models of
classroom walk-throughs for components such as purposes, participants, teacher
involvement, focuses or look-fors, protocols, data recording, and follow up. Kachur et al.
(2010) defined walk-throughs as “short, informal observations of classroom teachers and
students by school administrators, coaches, mentors, peers, and others, followed by
feedback, conversation, and/or action” (p. 1). Classroom walk-throughs serve to create a
snapshot of instruction and student learning to identify strengths, patterns, and needs of
an organization. The practice has been referred to as “learning walks, instructional walks,
focus walks, walk-abouts, data walks, learning visits, quick visits, mini-observations,
rounds, instructionally focused walk-throughs, administrative walk-throughs, supervisory
walk-throughs, collegial walk-throughs, reflective walk-throughs, classroom
walkthroughs, and just walk-throughs” (Kachur et al., 2010, p. 1).
Building on the work of Downey et al. (2004) and the Center for Comprehensive
School Reform and Improvement (2007) among others, Kachur et al. (2010) identified a
number of benefits to walk-throughs pertinent to the current study including “appraising
how professional development initiatives are being incorporated into classroom
practices” and “identifying professional development needs of the faculty and staff” (p.
8). Walk-throughs can be used to guide professional development via “peer coaching,
action research, data-driven decision-making, and group analysis of student work”
(Kachur et al., 2010, p. 15). Kachur et al. (2010) described several models of classroom
walk-throughs, some of which can be seen in Table 9.
The Learning WalkR Routine particularly focused on using walk-throughs to
“inform decisions about professional development,” especially “the extent to which new
practices and professional development content have been implemented in the classroom,
often pinpointing areas for further work” (Kachur et al., 2010, p. 31).
The Data-in-a-Day guidelines outlined by Kachur et al. (2010) can be applied to
all walk-through models. These included a) watching quietly while observing, b) locating
an unobtrusive place to sit and observe, and c) showing respect for teachers and students
during and after your visit, and d) acting in a way that will minimize impact on student
learning (Kachur et al., 2010, p. 66).
Table 9
Kachur, Stout, & Edwards (2010) Classroom Walk-through Models Matrix
Model Purpose Observers Look-fors
Visit/
Data
Frequency
Feedback
Instructional
Practices
Inventory
(IPI) Process
Increase student
engagement;
schoolwide data of
student engagement
for collaborative
study and problem-
solving
Teacher leaders; Six coding
categories principals, central on student
office staff; engagement;
observer
requires trained on coding
“certified” to data
collectors and
facilitators
1-3 minutes per How
majority classroom; each
of the students
classroom are engaged. observed
multiple
times in one day;
minimum 100
overall
Data provided for
analysis and
redesign of
instruction using
IPI process; steps
recommended for
study, reflection by
all faculty
The
Learning
Walk®
Routine
Inform decisions
about professional
development;
explore new
practices and content
from professional
development
Administrators or
teacher-leaders.
One or more of the
Institute’s Nine
Principles of Learning;
content-specific;
related to professional
development
5–25 minutes;
frequency
determined by
school
Open-ended
form for notes
about
evidence; for
participant use
only; teachers
are not
identified
Summary of
patterns and
reflective
questions,
followup
professional
development
topics, and
information on
next learning
walks.
Look2
Learning
(L2L)
(formerly
SMART
Walks)
Improve student
achievement;
analyzing data on
rigor, relevance, and
student engagement
Principals,
instructional
coaches, and/or
team leaders
Student learning,
engagement, and work;
indicators of learning,
curriculum alignment,
levels of thinking,
qualities of student
work, learner
engagement, and
instructional cycle.
Frequent 4minute
visits
Online form
using L2L
software
Cumulative,
anonymous data
summarized and
displayed
graphically to
analyze and reflect
on trends
Power
Walkthrough®
Extent to which
teachers use
Classroom
Instruction That
Works strategies;
technology use;
levels of Bloom’s
Taxonomy; evidence
of student learning
School and
district
administrators
and curriculum
directors and
teachers.
Classroom Instruction
That Works and
teacher and student use
of technology.
Visits are 3–5
minutes;
frequency
determined by
school
Online form to Power Walk-
build through™ reports;
reports/graphs share
observations of instructional with
teachers for activity to
coaching to higher
determine levels of extent to
which performance.
professional
development is evident
Teachscape
Classroom
Walkthrough
(CWT)
Improve student
achievement by
improving
instructional
practices
Instructional
leaders,
principals,
assistant
principals,
department
heads, coaches,
teachers
Research-based
practices in use of
questioning, student
engagement, categories
of effective
instructional strategies,
and degree of
differentiation
4–7 minutes;
frequency
determined by
school
Uses
Teachscape
software,
online data for
reports and
graphs
Teachers examine
the data to create
goal of improving
instruction and
hold reflective
discussions
Note: Adapted from Classroom Walk-throughs to Improve Teaching and Learning, by D.
Kachur, J. Stout, and C. Edwards. Published in 2010 by Eye on Education.
In addition to behavioral guidelines, Kachur et al. (2010) discussed various
methods of data gathering utilized by the 18 models studied. The group found there were
a variety of methods of collecting data ranging from no written notes to checklists to
detailed narratives or a combination of checklists and narratives. A checklist was utilized
when there were “specific look-fors that can be easily observed” (p. 91). The checklist
data recording method clarified “exact educational behaviors and activities expected”
(Kachur et al., 2010, p. 92).
Recommendations based on examining multiple walk-through models.
Narrative forms were utilized primarily for describing observations of teaching strategies
or teacher decisions during a walk-through. These included “classroom climate, personal
relationships, literacy support, and high expectations” (Kachur et al., 2010, p. 92).
Specific look-fors were also generally included on the form and agreed-to prior to
conducting walk-throughs. The combination form included both a checklist and space for
notes and was utilized to assist “managing productive conversations.” (Kachur et al.,
2010, p. 99).
Kachur et al. (2010) defined look-fors and focus questions as a method to frame
walk-throughs. Look-fors, according to Kachur et al. (2010), are “clear statements or
descriptors of observable evidence of teaching and learning such as specific instructional
activities, learning activities, behavioral outcomes, artifacts, routines or practices” (p. 76).
For the purposes of the current study, look-fors focused on the level of digital literacy
practices exhibited by teachers and students in a classroom.
Kachur, Stout, and Edwards (2013) continued this work in Engaging Teachers in
Classroom Walk-throughs. The group conducted a 2-year study of 40 schools from 30
school districts in 17 states and one district in Canada. The study focused primarily on
teacher involvement with the walk-through process. They advocated for teacher
involvement in the walk-through process due to a number of benefits including using
walk-throughs to complement professional development and for teachers to:
•Note useful practices other than the ones they use
•Ease the fear of trying something new
•Feel motivated to improve their craft
•Identify possible areas for their own professional development
•Identify areas of practice for reflective dialogue with colleagues
•Accelerate improvement in student performance (Kachur et al., 2013, p. 3).
Involving teachers in the walk-through process built collegiality among staff as
educators discussed practices, shared knowledge, and observed and supported each other
(Kachur et al., 2013). Recommendations for administrators to accomplish this were to:
•Solicit support of teacher leaders regarding walk-throughs as a form of
professional development.
•Involve teachers possessing positive influence over colleagues.
•Have teacher leaders participate early in the process and share their
experiences.
•Provide all teachers a safe method for addressing concerns.
•Emphasize the use of walk-throughs as a vehicle for professional growth
(Kachur et al., 2013).
During implementation, it was recommended that teachers were given input on
the selection of an established model or the design of the data gathering form and be
involved with establishing goals, protocols, and feedback. Recommended methods for
administrators to use to facilitate teacher involvement involved several strategies:
1. Utilize teacher leaders in communicating, advocating and implementing
schoolwide involvement.
2. Communicate a clear purpose of walk-throughs as a method of school
improvement.
3. Provide planning and a gradual, careful implementation.
4. Ensure transparency in the purpose, protocols, and expectations of
walkthroughs.
5. Create norms for the process and conversations.
6. Provide training for teachers on the process and value of selected walkthrough
model.
7. Schedule time for teachers to participate in and discuss the walk-throughs.
8. Encourage teachers to volunteer as participants.
9. Maintain focus on student learning.
10. Discuss data without “evaluative of judgmental comments” (Kachur et al.,
2013, p. 84).
A walk-through was defined by The Center for Comprehensive School Reform
and Improvement (2007) as a “brief, structured, nonevaluative classroom observation by
the principal that is followed by a conversation between the principal and the teacher
about what was observed” (p. 1). Various models of walk-throughs were examined by
Kachur et al. (2010) and made recommendations for look-fors and format of the
walkthrough document. These recommendations were considered during the construction
of the document produced during Round 4 of the current study. Table 10 contains a
summary of the literature reviewed regarding walk-throughs.
Table 10
Summary of Research on Walk-throughs
Author Date Purpose
Downey et al. 2004 Outlined Downey Walk-Through Process, benefits, how
to implement
Center for
Comprehensive
School Reform
and
Improvement
2007 Identified essential elements for a walk-through
Cervone &
Martinez-
Miller
2007 Made recommendations for implementing walk-through
Rossi 2007 Examined teacher perceptions regarding walk-throughs Sketta 2007
Recommended walk-throughs as a professional
development tool
Kachur et al. Compared 18 models of classroom walk-throughs;
made recommendations for designing and
implementing a walk-through process
Gillespie 2016 Identified attributes of successful walk-throughs
Cooper 2015 Identified components of an effective walk-through
document
Protheroe Outlined benefits of walk-throughs
Likert-type Scales
Likert scales were named for Robert Likert, who initially published a scale for
rating participant attitudes. “On this basis one of our cardinal problems is to find whether
social attitudes, in this sense, can be shown to be measurable, and if an affirmative
answer is forthcoming, a serious attempt must be made to justify the separation of one
attitude from others” (Likert, 1932, p. 8). Likert noted attitudes could be clustered and
placed on a continuum, which allows for a numeric value to be assigned. Thus,
participant attitudes about a social science topic may be measured.
Likert scales have been generally accepted to be ordinal as opposed to interval.
This was due to the fact that while the response choices on a Likert-type scale have an
order, the differences, or intervals, between those points may not be equal (Jamieson,
2004).
Bertram (2007) defined Likert-type scales as “A psychometric response scale
primarily used in questionnaires to obtain participant’s preferences or degree of
agreement with a statement or set of statements. Likert scales are a non‐comparative
scaling technique and are unidimensional (only measure a single trait) in nature” (p. 1).
Bertram also identified a number of strengths of Likert scales:
•easy to compile;
•likely to result in a highly reliable scale; and
•simple for participants to read and complete.
Bertram also noted a number of weaknesses for Likert scales:
•central tendency bias due to participants avoiding extreme responses;
•acquiescence bias in which participants aim to “please” the experimenter;
•social desirability bias in which participants attempt to present themselves in a
favorable light;
•lack of reproducibility; and
•validity may be difficult to demonstrate.
Carico and Perla (2008) addressed some of the debate between researchers
regarding Likert-type scales. There were two main schools of thought regarding the
character of Likert scales. One group determined the scales are ordinal, or produced rank
order data. The second group stated Likert-type scales are interval in nature. The
researchers concluded if a single item is being analyzed, it can be considered an interval
scale. However, this should occur “very rarely” (Carico & Perla, 2008, p. 1151).
It was appropriate, according to Carico and Perla (2008), to use means and
standard deviations when interpreting data. It was also “perfectly appropriate to calculate
Pearson correlation coefficients using the summative ratings from Likert scales and to use
these correlations as the basis for various multivariate analytical techniques, such as
multiple regression, factor analysis and meta-analysis, to obtain more powerful and
nuanced analyses of the data and research hypotheses being investigated” (Carico &
Perla, 2008, p. 1151).
Another debate among academics regarding Likert-type scales regarded the
number of options presented to participants. A Likert-type scale may contain as few as
four or as many as nine or more options. There was a particular divide on the use of a
midpoint option generally reading “no opinion” or “neutral” (Nadler, Weston, & Voyles,
2015). Nadler et al. (2015) conducted a study to examine the difference in participant
responses between a four-point Likert-type scale, a four-point Likert-type scale with a
“no opinion” option, and one with five options. The researchers also added a question to
qualitatively examine participant reactions to the mid-point item. While the three forms
of the questionnaire all obtained Cronbach alpha scores of 95% or more, reliability was
higher with the 5-point scale. Further, it was found the 5-point scale allows for a neutral
response, thus it was “closer to a true interval scale” (Nadler et al., 2015, p. 84).
Further research conducted by Willits, Theodori, and Luloff (2016) supported this
assertion and attempted to dispel several myths regarding Likert-type items. They
determined the middle option should not be deleted from response categories as
“undecided” was a valid response and may have forced participants to choose a side
regarding a topic on which they genuinely had no opinion. Willits et al. (2016) instead
recommended researchers examine characteristics of participants electing the “no
opinion” option. For the reasons identified above, a 5-point Likert-type scale was utilized
during the current study.
Willits et al. (2016) also addressed the debate as to whether questionnaires
utilizing Likert-type scales were to be classified as interval or ordinal. They found
participants “often view these as points on a continuum from low to high with response
distributions similar to those obtained on a scaled line with equal intervals between points
on that scale” (p. 134) and thus were to be treated as interval scales.
Likert scales were named for R. Likert, who initially published a scale for rating
participant attitudes. They are widely used in social science, medical, and educational
research. They were also the topic of debate as to the type of statistical analysis is
appropriate for the Likert-type scale items. There was also debate as to the optimal
number of options to include in a Likert-type scale. Findings from research about these
two debates were used in the construction and analysis of the Likert-type items utilized in
the current study. Table 11 summarizes the literature reviewed regarding Likert-type
scales.
Delphi Methodology
The Delphi method was first utilized by the Rand Corporation in the 1950s as
Project DELPHI. The first applications of the Delphi method were “to assess the direction
of long-range trends, with special emphasis on science and technology, and their probable
effects on society” (Gunaydin, n.d., p. 2).
Table 11
Summary of Research on Likert-type Scales
Author Date Purpose
Likert 1932 Created the Likert scale
Bertram 2007 Identified strengths and weaknesses of Likert scales
Carico & Perla 2008 Addressed debate as to whether Likert scales are
ordinal or interval in nature
Nadler et al. 2015 Studied mid-point item on Likert scales; determined it
was beneficial
Willits et al. 2016 Addressed debates of statistical analysis of
walkthrough data and mid-point option
The Delphi method was developed in a number of stages:
•secrecy and obscurity,
•novelty,
•popularity,
•scrutiny, and
•continuity.
The secrecy stage described the infancy of the Delphi method when the process
and results were classified by the military until the early 1960s. It was originally used by
military experts to reach consensus on sensitive problems. The second stage of the Delphi
method was novelty, which took place from the early 1960s until the late 1960s. The
Delphi method was primarily used at that time by corporations as a forecasting tool
(Yousuf, 2007).
The popularity stage was reached in the late 1960s and continued until the middle
of the 1970s when a variety of writings about the Delphi method appeared in print.
Scrutiny began in 1975 when the first critiques of the method appeared in print. The final
stage of development, continuity, is the current phase (Yousuf, 2007).
The classical Delphi technique study contained five features: “anonymity,
iteration, controlled feedback, statistical group response and stability in responses among
those with expertise on a specific issue” (Hanafin, 2004, p. 5). The purpose of the
arrangement was to allow the experts to reach consensus on a particular topic. The Delphi
technique aimed to meet these objectives: to decide upon a range of possible alternatives;
to explore underlying assumptions; to examine information that will lead to a consensus;
to correlate expert opinions from a variety of related disciplines; and to educate the
participants about the various aspects of the problem (Hsu & Sandford, 2007).
The controlled method of questioning in several iterations appeared to be “more
conducive to independent thought on the part of the experts and to aid them in the gradual
formation of a considered opinion” (Dalkey & Helmer, 1963, p. 459). Revisiting the
problem multiple times and examining answers provided by the group allowed each
expert to correct misconceptions and draw conclusions about various aspects of the
problem that he might not have considered in the initial round of questioning. It was
specifically designed for “participants to reassess their initial judgments about the
information provided in previous iterations” (Hsu & Sandford, 2007, p. 2). Responses
were computed in a statistical score after each round to represent the input from each
participant. The process was designed to eliminate pressure for conformity among group
members and to ensure equal opportunity for participation for all members (Fischer,
1978).
Helmer (1967), one of the initial Delphi method users, proposed three basic rules
for a successful Delphi application:
1. Select experts wisely.
2. Create favorable conditions for the experts to perform.
3. Use caution when deriving a single combined opinion from a varied group.
Examples of Studies Utilizing the Delphi Method
The Delphi method has been popular among educational researchers due to the
availability of current practitioners to serve as experts. Several Delphi studies pertaining
to education are described below.
Anderson (2007) conducted a study utilizing the Delphi method to identify
several factors regarding one-to-one laptop initiatives. The expert panel had three tasks:
1. Identify five strategies that will help superintendents overcome the barriers to
the implementation of one-to-one laptop programs in K-12 schools.
2. Determine the likelihood of implementation of these strategies in K-12
schools.
3. Determine the importance these strategies will have on the implementation of
one-to-one laptop programs in K-12 schools.
The study was completed with 26 superintendents and technology leaders with
experience in one-to-one initiatives. At the conclusion of three rounds, the expert panel
identified 11 strategies for superintendents to employ in order to overcome barriers to
implementing a one-to-one laptop program. Of those 11 strategies, it was determined they
all had a high level of importance; however, only four were likely to occur within the
given time frame of four years.
Verbeke (2014) conducted a Delphi method study to explore competencies
required for hiring an educational development leader. The four-round study utilized
experts in the field of educational development. The experts identified 66 competencies—
knowledge, skills, abilities, and values—required for a candidate in an educational
development leadership position. Seven of these traits failed to reach consensus for
inclusion on the list of competencies.
Verbeke (2014) also examined current job descriptions from across the United
States for correlation to these competencies. Upon examining the job descriptions, eight
additional competencies were identified that were not considered by the Delphi panel.
Casstevens (2016) utilized the Delphi method to create standards for joint-use libraries in
Texas. A joint-use library was defined as one that functions as both a school and a public
library. A panel of 29 library experts determined categories and components critical to a
joint-use library. These included objectives, goals, and principles that evolved into 42
standards in the course of the three-round study. From these standards, a checklist was
also developed for joint-use library self-assessment.
Vankykgibson (2016) conducted a modified Delphi study of 22 expert panelists to
identify strategies and guidelines necessary to an effective educational technology plan.
The three-round Delphi study initiated with 56 statements based on current research.
From these statements, an instrument was created to serve as a model for implementing
educational technology plans. Panelists also reached consensus about the roles and
responsibilities of the state to support and guide districts when implementing educational
technology plans.
The Delphi method has been popular among educational researchers due to the
availability of current practitioners to serve as experts, as demonstrated by the four
studies outlined in the review of literature contained in this chapter. Table 12 summarizes
the literature reviewed on the Delphi method.
Table 12
Summary of Research on Delphi Method
Author Date Purpose
Yousuf 2007 Outlined history of Delphi method and various types of
Delphi methods
Hanafin 2004 Described features of a successful Delphi study,
strengths and weaknesses
Hsu &
Sandford
2007 Discussed objectives of Delphi method, made
recommendations for a successful study utilizing
Delphi method
Dalkey &
Helmer
1963, 1967 Proposed basic rules for Delphi and method for
questioning
Anderson 2007 Conducted Delphi study to identify factors regarding
one-to-one laptop initiatives
Verbeke 2014 Conducted Delphi study to identify competencies
required for hiring an educational development leader
Casstevens 2016 Conducted Delphi study to identify standards for
jointuse libraries and self-evaluation of joint-use
libraries
Vankykgibson 2016 Conducted Delphi study to identify strategies and
guidelines for an effective educational technology plan
Summary
Digital citizenship is the “norms of appropriate, responsible behavior with regard
to technology use” (Ribble & Bailey, 2004, p. 1). Digital citizenship describes the actions
of a participant in a digital environment. Digital literacy was defined as “the ability to use
information and communication technologies to find, understand, evaluate, create, and
communicate digital information, an ability that requires both cognitive and technical
skills” (ALA, 2013, p. 2). Application of digital literacy skills would encompass
“troubleshooting abilities, purposes for connecting, skills in using popular technology
tools, as well as communication literacy, and web literacy” (Blummer, 2008, p. 38).
Information literacy is “the process of recognizing information need, finding,
evaluating and using information to acquire or extend knowledge” (Behrens, 1994, p. 33).
Internet literacy is “the capability to access and evaluate online information,” online
search competence, and “skill with connectivity, security, communication, multimedia,
and web page development” (Johnson, 2007, p. 434). These literacies are all interwoven
and required skills overlap. Research has demonstrated they all are crucial to students as
they learn to navigate a global, connected world.
Two of the most common technology integration models, the SAMR model
developed in 2006 and the TPACK in 1986, were described in this literature review. The
research in each instance indicates that teachers need additional support and professional
development in order for teachers to integrate technology at a higher level.
A walk-through was defined by The Center for Comprehensive School Reform
and Improvement (2007) as a “brief, structured, nonevaluative classroom observation by
the principal that is followed by a conversation between the principal and the teacher
about what was observed” (p. 1). Kachur et al. (2010) examined various models of
walkthroughs and made recommendations for look-fors and format of the walk-through
document.
Likert scales are widely used in social science, medical, and educational research.
They are also the topic of debate as to the type of statistical analysis is appropriate for the
Likert-type scale items. There is also debate as to the optimal number of options to include in
a Likert-type scale. Findings from research about these two debates were used in the
construction and analysis of the Likert-type items utilized in the current study. The Delphi
method was first utilized by the Rand Corporation in the 1950’s as Project DELPHI. The
classical Delphi technique study contained five features: “anonymity, iteration, controlled
feedback, statistical group response and stability in responses among those with expertise on a
specific issue” (Hanafin, 2004, p. 5). The purpose of the arrangement was to allow the experts
to reach consensus on a particular topic. The Delphi method has been popular among
educational researchers due to the availability of current practitioners to serve as experts, as
demonstrated by the four studies outlined in the review of literature contained in this chapter.
Recommendations for conducting successful Delphi method research were followed during
the process of the current study. Chapter 3 describes the Delphi method in greater depth.
CHAPTER 3. METHODOLOGY
While students have grown up online, in current digital citizenship education, they
are not being provided the skills needed to use digital information productively, especially
in an academic environment (Sharkey, 2013). A lack of digital literacy competency in
teachers may be exacerbating the problem. Compounding the issue is that the components
of digital literacy are often not explicitly taught to teachers during preservice or in-service
trainings. Teachers are left to fend for themselves should they choose to develop digital
competency. Few districts offer professional learning on digital literacy-related topics.
These topics would include skills such as the use and evaluation of digital tools;
leveraging advanced search options to limit and refine internet searches; and assessing
whether an online resource or person is credible and trustworthy.
A review of the literature on digital literacy produced information on assessing
student digital literacy competence and on teacher perceptions of digital literacy, but
failed to produce a tool for assessing the digital literacy competency of classroom
teachers. In many cases, administrators use the colloquial definition of “I know it when I
see it” to identify effective digital literacy practices demonstrated by educators.
The purpose of the current study was to examine the behaviors teachers exhibit
that are integral to teaching digital literacy in order to create a definition of the digitally
literate teacher and a non-evaluative walk-through document. The definition attempted to
clarify the ambiguous definitions as presented in the literature review. The walk-through
document provides administrators or instructional technology staff a tool for measuring
the digital literacy proficiency of teachers and a data source for guiding professional
development decisions.
The objectives of the current study were to conduct a Delphi method study of
experts in the digital literacy field. Using a Delphi method, the list of observable teacher
behaviors that exemplify a high level of competence in digital literacy instruction were
then formatted into a brief, non-evaluative walk-through document and a definition of a
digitally literate teacher. The current study was conducted in four iterations, the first of
which identified the teacher behaviors, the second and third consisted of ratings using a
Likert-type scale to determine which items most needed to be in the definition and on the
walk-through document, and the fourth round provided an opportunity for participants to
provide input on a sample definition and walk-through document.
Research Questions
The current study created a definition of a digitally literate teacher and a
nonevaluative document identifying possible areas of digital literacy deficiency in
teachers and areas at which teachers excel. The research questions this study addressed
were as follows:
Research Question 1 (RQ1) What are the indicators of digital literacy competency in
teachers as identified by a panel of experts?
Research Question 2 (RQ2) Which of these behaviors could be observed on a
walkthrough observation?
Research Question 3 (RQ3) What are the elements of such a form?
Research Design
The current study utilized the Delphi method. The Delphi technique was named
for the ancient Greek oracle at Delphi who “offered visitations of the future for those
seeking advice” (Hanafin, 2004, p. 4). The oracle was one who has “unquestioned
wisdom and knowledge or of infallible authority” (Yousuf, 2007, p. 1). The Delphi
method has roots in the philosophy of educator John Dewey who believed that social
science research should directly impact real-world practice and decision-making (Brady,
2015).
The Delphi method is particularly useful in the following situations:
•The problem does not lend itself to analytical techniques but can benefit from
“subjective judgements on a collective basis” (Hanafin, 2004, p. 9).
•The experts on a particular subject are “In different fields and occupations and
not in direct communication” (Hanafin, 2004, p. 9).
•“The number of specialists is too large to effectively interact in a face-to-face
exchange and too little time and/or funds are available to organize group
meetings” (Hanafin, 2004, p. 9).
The popularity of Delphi studies has spawned several different types of Delphi
methods, which are dependent upon the desired outcome of a study. A modified Delphi
contains face-to-face interviews or focus groups in the first round. A policy Delphi is used
to analyze policy issues. The goal of a policy Delphi is not to reach consensus; it is to
understand a variety of viewpoints pertaining to the policy at hand. (Davidson, 2013).
The emphasis is to identify different opinions via debate throughout the rounds (Yousuf,
2007).
A normative Delphi is designed to establish goals and priorities as opposed to
forecast. A decision Delphi attempts to gather decision-makers to generate a decision
about future developments. A real-time Delphi occurs online where panelists visit a
webpage simultaneously. Another version of the real-time Delphi is the technological
Delphi where panelists use a hand-held device to respond immediately to questions posed
by the researcher. Finally, a disaggregative Delphi involves disaggregating responses of
panelists via cluster analysis (Davidson, 2013). The traditional Delphi format was used
for the current study.
One distinct advantage of the Delphi technique was that it offered the opportunity
to “recognize and acknowledge the contribution of each participant” (Hanafin, 2004, p.
8). Other advantages included maintaining participant anonymity to prevent participants
influencing the opinions of others and ensuring the statements were more likely to be
based in personal experience as opposed to experts taking a popular or institutional stance
on a topic. The process also enabled panelists who were particularly vocal or reticent in
face-to-face meetings to have equal participation during the conversation, which
minimized the possibility of coercion or manipulation. (Hsu & Sandford, 2007). The
utilization of written surveys also allowed for expert participation from a wide
geographical area. Other benefits of the Delphi method included its flexibility that may be
applicable in both quantitative and qualitative research and that Delphi studies have had a
practical purpose in guiding practice, policy, or decision making (Brady, 2015).
Disadvantages of the Delphi technique were due mainly to the anonymity of
participants. The anonymity may have led to “a lack of accountability because responses
may not be traced back to the individual” (Hanafin, 2004, p. 11). There was also the
disadvantage to the aspect of consensus, which may have allowed for a diluted opinion
and focus on the lowest common denominator (Hanafin, 2004). The method has also been
criticized for being time and labor intensive.
Another disadvantage was the possibility that preconceptions of the panelists may
have led to an inability to consider other perspectives during a Delphi study (Yousuf,
2007). Further limitations included the fact that the process of achieving consensus may
have led to dismissing divergent ideas and may have led to a false consensus.
Criticism of the Delphi method includes that the method is
a) being unscientific; b) having a low level reliability if judgments among experts
and therefore dependency of forecasts on the particular judges selected; c) the
sensitivity of results to ambiguity in the questionnaire that is used for the data
collection in each round; and d) the difficulty in assessing the degree of expertise
incorporated into the forecast. (Yousuf, 2007, p. 5)
Some dispute that the limitations were inherent in the method but that poor results were
from a poor application.
One caveat to utilizing the Delphi method in the fields of science or technology is
that change occurs so rapidly that “forecasting the degree of uncertainty is so great that
exact and always correct predictions are impossible, so a high degree of error is to be
expected” (Yousuf, 2007, p. 6). Another caveat is that technology-related developments
are not always predicted by inside experts but may have occurred in the form of
disruptive innovations by outside parties.
Setting
Because of the nature of a Delphi technique as an assemblage of a variety of
experts, the setting for the current study was international. The expert panelists
represented nine states and three countries including the United States. Ten of the
participants were from Texas, followed by three in Maryland, and three from
Massachusetts. Two participants were in Illinois. One participant each represented
Alabama, Kansas, Kentucky, Michigan, New York, and Pennsylvania. One participant
was in Cairo, Egypt and one was in Singapore.
Communication was made via electronic communication methods such as email,
online documents, and web-based surveys. The experts were able to participate using
their own technology in a setting that best suited their schedule.
Participants
There are four requirements for expertise in a Delphi study: a) knowledge and
experience with the issues under investigation; b) capacity and willingness to participate;
c) sufficient time to participate in the Delphi; and, d) effective communication skills
(Skulmoski, Hartman, & Krahn, 2007). A list of participants from national and
international organizations who were likely to demonstrate the criteria above was created
by the researcher in the form of a knowledge resource nomination worksheet (Okoli &
Pawlowski, 2004).
Experts in the fields of instructional technology, school administration,
professional development, and school librarianship were polled for participation. An
expert for the purposes of the current study was defined as an educator who has worked
in their field for at least 10 years and has demonstrated competence by publishing,
presenting, or actively participating in professional organizations at the state, national, or
international level on the topics of web literacy, information literacy, digital literacy,
professional learning for instructional technology implementation, and/or digital
citizenship.
Selection of Participants
Participants were invited to participate on the basis of the criteria listed above. As
a member of several professional organizations such as the Consortium for School
Networking (CoSN), the American Association of School Administrators (AASA), the
International Society for Technology in Education (ISTE), and the American Library
Association (ALA), the researcher had access to a network of national and international
experts who were also members of these organizations. Expert participants were selected
from membership in those groups.
The experts were contacted via email or online announcement in the ISTE
Administrator Network and the CoSN webpages to gauge their willingness to participate
in the current study. These experts were asked to complete an online form to provide
contact information and to document their expertise in the area of digital literacy. A copy
of the online form can be found in Appendix A. A minimum of 30 participants, from the
fields of educational administration, instructional technology administration, and library
media services administration, was the desired group size. A total of 38 volunteers
completed the online form indicating their interest in participating in the current study.
Data Collection
The Delphi technique typically contains at least three rounds of questionnaires to
generate consensus or a sufficient statistical analysis of responses. The current study
utilized four rounds. The rounds were broken down as follows during the data collection
phase.
Prior to Round 1
The first step to conducting the current Delphi method survey was to determine if
the aim of the study was to “measure diversity of opinions on a topic or to steer a group
towards consensus” (Iqbal & Pipon-Young, 2009, p. 598). Several questions were
answered to determine if the Delphi method was appropriate:
•What kind of group communication process is desirable in order to explore the
problem at hand?
•Who are the people with expertise on the problem and where are they located?
•What are the alternative techniques available and what results can reasonably
be expected from their application?” (Gunaydin, n.d., p. 4).
The second step was to determine the estimated number of rounds and develop a
timeframe for the current study. Because the aim of the current study was to reach
consensus on a subject, three or more rounds were preferable (Iqbal & Pipon-Young,
2009). The current study was completed in four rounds. If the study had been intended to
measure opinions about a topic, it may have been conducted in two rounds. It was
estimated that a four round Delphi study required approximately 10 to 12 weeks to
implement (Dennington & Storm, 2006).
Step 3 consisted of composing the initial research questions. The research
questions were open-ended, qualitative questions pertaining to the research topic. The
first round of the Delphi involved “a series of open-ended questions inviting panelists to
brainstorm” (Iqbal & Pipon-Young, 2009, p. 599). These questions were generated after a
thorough literature review of the relevant topics. A reasonable amount of time to complete
the initial round of questioning of 30 minutes was given as recommended by Iqbal and
Pipon-Young (2009).
Step 4 of the Delphi method involved selecting panelists to participate in the
current study. A list of criteria for determining expertise was created. Expertise was
defined for the current study as an educator who has worked in their field for at least 10
years and has demonstrated competence by publishing, presenting, or actively
participating in professional organizations at the state, national, or international level on
the topics of web literacy, information literacy, digital literacy, professional learning for
instructional technology implementation, and/or digital citizenship.
Research recommended that a diverse group of experts was utilized to provide the
greatest level of reliability to the study and to encourage a variety of initial responses. For
the current study, educators and administrators at both the campus and central office
levels were chosen as expert panelists. Researcher opinions differed on the size of the
expert panel required. Delphi studies have been conducted with as few as four and as
many as 1,000 panelists (Iqbal & Pipon-Young, 2009). A panel ranging in size from 10 to
50 was recommended when considering the amount of data and analysis required for the
study. A total of 38 volunteers completed the online form indicating willingness to
participate in the current study.
Step 5 involved contacting the experts and inviting them to participate in the study
via face to face conversations and emails. An online announcement was also posted on
the ISTE Administrator Network and the CoSN webpages to solicit experts likely to meet
the pre-determined criteria. An explanation of the goals and subject of the current study,
criteria for participation, and estimated time requirements accompanied the invitations.
Participants were asked to complete an online form to collect basic demographic and
contact information at the time they agreed to participate in order to determine whether
they possessed the required qualifications.
Round 1
During Step 6, panelists were emailed a link to a survey hosted on the
Surveymonkey.com website. The survey contained the informed consent document,
demographic questions, and three open-ended questions. Panelists were also given
information regarding the purpose of the current study. It was imperative that panelists
understood the aim of the current study to prevent answering questions inappropriately or
losing interest in the study (Gunaydin, n.d.). Follow-up was conducted via deadline
reminders and individual thanks for completing the questionnaire. Of the 38 participants
receiving the invitation to participate, 26 completed the Round 1 questionnaire, a 68%
participation rate. This was just shy of the 70% participation rate deemed acceptable by
Iqbal and Pipon-Young (2009).
The first round of the current study began with an online questionnaire exploring
what it means to be a digitally literate teacher. The initial online questionnaire presented
to participants was designed to elicit general discussion about digital literacy in the
classroom. The Round 1 survey is shown in Appendix A. Three open ended questions
were asked:
Delphi Question 1 (DQ1) What qualities does a teacher possess and demonstrate that
indicate a high proficiency in digital and web literacy?
Delphi Question 2 (DQ2) What are the observable behaviors of a teacher with a high
level of digital literacy competence?
Delphi Question 3 (DQ3) What are the observable behaviors of students in a classroom
with a teacher demonstrating a high level of digital literacy competence? The informed
consent document and brief biographical survey was also administered to compile
demographic information about the participants.
Round 1 was designed to solicit specific information about the content area.
Respondents had 3 weeks to reply to the current online questionnaire. At the close of the
3 weeks, the researcher compiled the responses and created a survey instrument for the
second round of questioning.
Responses gathered in Step 6 were then analyzed in Step 7. Identifying data was
removed from the responses to ensure anonymity and each panelist received a number
assigned by the Surveymonkey.com website. Responses were summarized for common
and divergent themes and categorized to identify common elements. Duplicate responses
were noted, and duplicate items were eliminated. The remaining items were utilized to
generate the second round of questions for the Delphi study.
Step 8 involved the creation of the second-round questionnaire based on the
responses from the first round. The second and third rounds of the current study were
quantitative and involved a Likert-type scale for panelists to rate appropriateness of each
item for inclusion in the definition and the walk-through document.
Round 2
The second round of the current study began with Step 9. Participants who
completed and returned the questionnaire from the first round were emailed the new
survey and a summary of the responses generated during Round 1. Similar to Round 1,
they were also provided a deadline for responses and individualized contact during the
time allotted for completing the second questionnaire.
The second round of the current study presented participants with a survey
instrument created from the responses received in the first round and a summary of
responses from all participants. These can be found in Appendices B and C. The survey
included a list of observable behaviors obtained from the first round that participants
were asked to rank using a Likert-type scale of 1-5.
The Likert-type scale differed slightly for DQ1 than for DQ2 and DQ3. RQ1 was
designed to determine elements to be included in the definition of a digitally literate
teacher. The rating scale for DQ1 was as follows:
1 = Should not be included in the definition
2 = Unimportant for use in the definition
3 = Depends on the circumstances if the current item should be included in the
definition
4 = Important to include in the definition
5 = Essential to include in the definition
DQ2 and DQ3 were intended to correspond to RQ2. RQ2 aimed to generate a list
of observable classroom behaviors in order to create a walk-through form to determine
the digital literacy proficiency level for a teacher. The Likert-type scale used for DQ2 and
DQ3 was as follows:
1 = Should not be included in the walk-through document
2 = Unimportant for use in the walk-through document
3 = Depends on the circumstances if the current item should be included in the
walk-through document
4 = Important to include in the walk-through document
5 = Essential to include in the walk-through document
The 10th step of the Delphi study involved a statistical analysis of the data
generated during the second round of questioning. The mean, median, mode, and standard
deviation of each item were calculated using the IBM SPSS software to determine
consensus. Internal consistency of the questionnaire was measured via a Cronbach’s alpha
analysis.
For the purposes of the current study, consensus on an item met one or more of
the following criteria in order to be included in the definition or the walk-through
document:
•70% or greater panel agreement;
•4.0 or greater mean score;
•4.0 or greater median and mode score; or
•1.0 or less standard deviation score.
Consensus on an item met one or more of the following criteria in order to be
eliminated from the definition or the walk-through document:
•70% or greater panel agreement;
•2.99 or less mean score;
•2.99 or less median and mode score; or
•1.0 or less standard deviation score.
The items that failed to reach consensus for the definition or walk-through
document were used to generate the survey for the third round.
Round 3
In Step 11, the researcher generated the questionnaire for the third round of the
Delphi study. A summary of the responses from Round 2 was presented to participants for
input during Round 3 on an online document shown in Appendix D. Data generated
during the second round was utilized to generate questions regarding items that had not
yet reached consensus. Those items were used to generate a questionnaire for the third
round to provide panelists an opportunity to reevaluate the items for inclusion in the
definition or walk-through document.
Step 12 involved communicating results to the participants from Round 2. Results
were reported for items that had reached a pre-determined consensus and participants
were also given the criteria used to determine consensus. Participants also received a
survey from the Surveymonkey.com website to complete in order to reevaluate the items
failing to reach consensus. A copy of that survey is found in Appendix D. Participants
were given a 10-day window to respond to the items using the same Likert-type scale
used during Round 2. Again, individualized messages to panelists in the form of reminder
and appreciation emails occurred throughout Round 3.
Step 13 involved a statistical analysis of the data generated from the third round in
a similar manner outlined during Step 10. Mean, median, mode and standard deviation for
items in Round 3 were calculated using SPSS software. Internal consistency of the third-
round questionnaire was measured using a Cronbach’s alpha analysis. Items reaching
consensus for inclusion in the definition or walk-through document were determined as
were the items reaching consensus for elimination.
For the purposes of the current study, consensus on an item met one or more of the
following criteria in order to be included in the definition or the walk-through document:
•70% or greater panel agreement;
•4.0 or greater mean score;
•4.0 or greater median and mode score; or
•1.0 or less standard deviation score.
Consensus on an item met one or more of the following criteria in order to be
eliminated from the definition or the walk-through document:
•70% or greater panel agreement;
•2.99 or less mean score;
•2.99 or less median and mode score; or
•1.0 or less standard deviation score.
Items failing to reach consensus after two iterations of ranking on a Likert-type
scale were reported to panelists along with the items that reached consensus. In order to
prevent fatigue in participants, items failing to reach consensus at the close of Round 3
were not deemed essential and were therefore not included in the definition and the
walkthrough document.
It has been recommended that two or three rounds of a Delphi study are sufficient
to prevent participant fatigue (Hsu & Sandford, 2007; Verbeke, 2014). The current study
was completed in three iterations or four rounds.
Round 4
When consensus had been reached on the essential elements for a definition and
the look-fors for the walk-through form, the mock-up of the definition and the form was
sent to participants during Round 4. For the definition, panelists were given the following
options:
•Accept the definition of a digitally literate teacher as written
•Do not accept the definition of a digitally literate teacher as written
•Accept the definition of a digitally literate teacher with modification.
Panelists were given similar options after reviewing the proposed walk-through
document. A third question, “Who do you think would use this document?” was also
presented to panelist with the following options:
•Campus principal or assistant principal
•Central office administrator
•Instructional coach, specialist, facilitator
•Other (please specify).
At the end of the 10-day window, the researcher thanked the expert panelists for
their participation and compiled the responses to the three questions from the Round 4
questionnaire.
Role of the Researcher
The researcher in the current study was an administrator in the digital learning
department of a large suburban school district in North Texas. The researcher was
responsible for the supervision of approximately 70 library media specialists and
instructional specialists. The district had been implementing a 1:1 initiative for
approximately 5 years when the current study was initiated. Members of the digital
learning department were responsible for providing instructional and technological
integration support and training to the students and teachers in the district. Under the
leadership of the researcher, the library media specialists and instructional specialists
were 4 years into the process of creating and teaching a district-wide digital citizenship
and digital literacy curriculum. The researcher has presented about digital citizenship and
digital literacy at the state, national, and international levels and has published articles
related to these topics in national trade journals. The researcher met the criteria for
participation as an expert in the current Delphi method survey and thus was qualified to
select expert participants from colleagues in related fields. The researcher interacted with
the participants of the current survey about the current study only via electronic means to
prevent communicating bias or assumptions.
Other Data Sources
Other data that may have factored into the Delphi method were resources
available to participants online such as national and international instructional technology
standards and sample assessments for student digital literacy competence. These sources
would have been used primarily as resources were the participants to decide to evaluate
them independently. Participants may have also examined walk-through documents on
other topics to determine format and provide input on items regarding the document
produced during the current study.
Data Analysis
Responses from the second and subsequent rounds of the process were analyzed
to calculate the mean, median, mode, and standard deviation to determine items that have
reached consensus. Internal consistency of the questionnaires presented to expert
panelists in Rounds 2 and 3 were also calculated using a Cronbach’s alpha analysis. For
the purposes of the current study, consensus on an item met one or more of the following
criteria in order to be included in the definition or the walk-through document:
•70% or greater panel agreement;
•4.0 or greater mean score;
•4.0 or greater median and mode score; or
•1.0 or less standard deviation score.
Consensus on an item met one or more of the following criteria in order to be
eliminated from the definition or the walk-through document:
•70% or greater panel agreement;
•2.99 or less mean score;
•2.99 or less median and mode score; or
•1.0 or less standard deviation score.
The items that reached consensus at the end of three rounds were included on the
final definition and non-evaluative walk-through document that created as a result of the
current study.
Verification
Data was verified with participants via use of secure online forms the participants
were able to access at any time throughout the current study. Study protocols, guidelines,
and summaries of questionnaire answers were noted in the online forms. Each iteration of
the Delphi method included a summary of findings from the previous round so
participants were able to ask questions or clarify previous statements. Participants were
also provided with the final document for input or comments and to verify the accuracy
of content.
Ethical Consideration
Confidentiality of participants was protected during the current study by assigning
each participant a number through the online survey instrument website. Participants
were given adequate time to respond during the iterations of the Delphi process and were
able to compile responses on the electronic survey instruments and questionnaires in the
setting of their choosing. The current study was conducted according to procedures
outlined by the Dallas Baptist University Committee for the Protection of Human
Participants, and the researcher received permission from the committee to conduct the
current study. The current study also conformed to the United States Department of
Health and Human Services Regulations for the Protection of Human Subjects.
CHAPTER 4. RESULTS
A New Media Consortium Horizon survey of more than 450 education leaders,
faculty, and staff conducted in 2016 found educators do not have a clear understanding or
shared definition of digital literacy, which impedes the ability to instruct students on
digital literacy skills. Furthermore, administrators have no tool to use to evaluate the
digital literacy competency of classroom teachers.
There appears to be a gap in the literature describing demonstrable behaviors of
teachers possessing a high competence of digital literacy. There is also a gap in the
literature about professional development for teachers, other than self-paced online
courses, to hone digital literacy skills.
Suppo (2013) specifically addressed the need for “a tool to measure the level of
digital citizenship demonstrated daily by administrators, teachers, and students” (p. 96) as
a conclusion of his study. Hatlevik et al. (2014) recommended studies “to describe good
teacher roles for fostering students’ digital competence” (p. 229). The current study
attempted to address the gap in the literature as described in these two studies as well as
the additional research in digital literacy advised in the findings of other studies (Ba et al.,
2002; Jones & Mitchell, 2015).
The purpose of the current Delphi study is twofold. The first purpose was to
utilize a panel of expert practitioners to identify the characteristics of a teacher with a
high level of digital literacy competency because there was not a widely agreed-upon
definition based on research conducted by the researcher. The second purpose was to
examine observable teacher behaviors integral to teaching digital literacy in order to
create a non-evaluative walk-through document for measuring the digital literacy
proficiency of teachers.
The current study utilized a four-round Delphi method to ascertain the traits to
compose the definition of a digitally literate teacher and the demonstrable behaviors of
students and the teacher in a classroom with a teacher possessing a high degree of digital
literacy. Chapter 4 presents the results from four rounds of survey questions and the
analysis of the data provided from the results. The research questions below guided the
current study.
Research Questions
The current study sought to create a definition of a digitally literate teacher and a
non-evaluative document identifying possible areas of digital literacy deficiency in
teachers and areas at which teachers excel. The research questions the current study
addressed are as follows:
Research Question 1 (RQ1) What are the indicators of digital literacy competency in
teachers as identified by a panel of experts?
Research Question 2 (RQ2) Which of these behaviors could be observed on a
walkthrough observation?
Research Question 3 (RQ3) What are the elements of such a form?
Delphi Methodology
The objectives of the current study were to conduct a Delphi method study of
experts in the digital literacy field, particularly as it pertained to web literacy. Using a
Delphi method, the list of observable teacher behaviors that exemplify a high level of
competence in digital literacy instruction were then formatted into a definition of a
digitally literate teacher to address RQ1 and a brief, non-evaluative walk-through
document to address RQ3.
The Delphi method is particularly useful in the following situations:
•The problem does not lend itself to analytical techniques but can benefit from
“subjective judgements on a collective basis” (Hanafin, 2004, p. 9).
•The experts on a particular subject are “In different fields and occupations and
not in direct communication” (Hanafin, 2004, p. 9).
•“The number of specialists is too large to effectively interact in a face-to-face
exchange and too little time and/or funds are available to organize group
meetings” (Hanafin, 2004, p. 9).
The popularity of Delphi studies has spawned several different types of Delphi
methods, which were dependent upon the desired outcome of a given study. A modified
Delphi contains face-to-face interviews or focus groups in the first round. A policy Delphi
is used to analyze policy issues. The goal of a policy Delphi is not to reach consensus; it
is to understand a variety of viewpoints pertaining to the policy at hand. (Davidson,
2013). The emphasis is to identify different opinions via debate throughout the rounds
(Yousuf, 2007).
A normative Delphi is designed to establish goals and priorities as opposed to
forecast. A decision Delphi attempts to gather decision-makers to generate a decision
about future developments. A real-time Delphi occurs online where panelists visit a
webpage simultaneously. Another version of the real-time Delphi is the technological
Delphi where panelists use a hand-held device to respond immediately to questions
posed by the researcher. Finally, there is a disaggregative Delphi in which the panelist
responses are disaggregated via cluster analysis (Davidson, 2013).
The traditional Delphi format was used for the current study. The traditional, or
classical, Delphi technique study contains five features: “anonymity, iteration, controlled
feedback, statistical group response and stability in responses among those with
expertise on a specific issue” (Hanafin, 2004, p. 5).
Selection of Expert Panel
An expert panel was selected using the following criteria—10 or more years as an
educator and demonstrated competence through publishing, presenting, or actively
participating in professional organizations at the state, national, or international level on
the topics of web literacy, information literacy, digital literacy, professional learning for
instructional technology implementation, and/or digital citizenship. Thirty-eight
volunteers completed the online form agreeing to participate in the current study. All were
able to demonstrate suitability as an expert according to the criteria listed above.
Expert Panel Size and Demographics
Of the 38 volunteers invited to participate in the current study, 26 of them
responded to the invitation. Of these 26 expert panelists completing Round 1, 69% of
them were female, 31% were male. The most common panelist age range was between 50
and 59 years old with nine panelists indicating this. The second most common age range
was between 40 and 49 years old with seven panelists falling in this range. Six panelists
were between 60 and 69 years old and four were between 30 and 39 years old, as
demonstrated in Figure 3.
Figure 3. Age range of panelists.
20-29 30-39 40-49 50-59 60-69 70+
0
1
2
3
4
5
6
7
8
9
10
Age of Expert Panelists
Fifteen of the panelists obtained a master’s degree level of education. Ten of the
panelists earned doctorates. Only one of the panelists reported a bachelor’s degree as the
highest level of education earned, as demonstrated in Figure 4. One panelist noted that an
Educational Specialist degree had been earned in addition to a master’s degree, but that
level was not included as an option in the demographic question.
The panelists represented nine states and three countries including the United
States. Ten of the panelists were from Texas, followed by three in Maryland, and three
from Massachusetts. Two panelists were in Illinois. One panelist each represented
Alabama, Kansas, Kentucky, Michigan, New York, and Pennsylvania. One panelist was
in Cairo, Egypt and one was in Singapore.
Figure 4. Panelist education level.
Bachelor
Masters
Doctorate
Panelist Education Level
The panelists reported a combined 663 years of experience in education. Thirteen
panelists had between 20 and 29 years of experience. Six panelists had between 30 and 39
years of experience. Five panelists had from 10 to 19 years of experience, and two had
been educators for over 40 as shown in Figure 5.
Panelists reported expertise in one or more areas including instructional
technology/technology, educational administration, and library media services.
Instructional technology/technology was indicated as an area of expertise by 21
participants. Twenty panelists indicated expertise in educational administration and 10
panelists indicated expertise in library media services. Sixteen of the panelists served in a
leadership position related to technology such as a director or chief technology officer.
Figure 5. Panelist experience in years.
Three of the panelists were campus-level administrators. Three panelists were in
district-level leadership positions related to library media services. Two panelists were
employed as district-level administrators in a curriculum or professional development
capacity. Two panelists were college professors.
0
2
4
6
8
10
12
14
10-19 20 - 29 30 - 39 40+
Panelist Experience in Years
Consensus
Responses from the second and third rounds of the current Delphi study were
analyzed for measures of central tendency including mean, median, mode and standard
deviation. These calculations were used to determine items that reached consensus. For
the purposes of the current study, consensus on an item met one or more of the following
criteria in order to be included in the definition or in the walk-through document:
•70% or greater panel agreement;
•4.0 or greater mean score;
•4.0 or greater median and mode score; or
•1.0 or less standard deviation score.
Consensus on an item met one or more of the following criteria in order to be
eliminated from the definition or from the walk-through document:
•70% or greater panel agreement;
•2.99 or less mean score;
•2.99 or less median and mode score; or
•1.0 or less standard deviation score.
Description and Results of Delphi Rounds
The current study was conducted in four iterations, the first of which identified the
teacher behaviors, the second and third consisted of ratings using a Likert-type scale to
determine which items most needed to be in the definition and on the walk-through
document, and the fourth round provided an opportunity for participants to provide input
on a sample definition and walk-through document. A timeline of the four rounds is found
in Table 13.
Table 13
Timeline of Delphi Study
Iteration Date Opened Date Closed
Round 1 December 10, 2017 January 3, 2018
Round 2 January 15, 2018 January 26, 2018
Round 3 February 19, 2018 March 5, 2018
Round 4 March 27, 2018 April 6, 2018
Delphi Round 1
Members of the expert panel were provided a link to a survey instrument
including an informed consent option, questions about demographic information, and
three open ended questions.
Timeline for Round 1
The online survey was sent via email on December 10, 2017 to 38 experts who
had previously indicated an interest in participating in the current study. The survey,
hosted on the SurveyMonkey.com website, contained a mandatory informed consent
response, eight basic demographic questions, and three open-ended questions. The survey
can be found in Appendix A. The open-ended questions are as follows:
Delphi Question 1 (DQ1) What qualities does a teacher possess and demonstrate that
indicate a high proficiency in digital and web literacy?
Delphi Question 2 (DQ2) What are the observable behaviors of a teacher with a high
level of digital literacy competence?
Delphi Question 3 (DQ3) What are the observable behaviors of students in a classroom
with a teacher demonstrating a high level of digital literacy competence?
On December 22, 2017, a reminder email was sent to volunteers who had not yet
participated in the current study and a thank you email was sent to those who had
completed responses. Due to replies regarding their schedules prior to the holiday season,
the end date for Round 1, initially set for December 24, 2017, was extended to January 3,
2018.
Round 1 Analysis and Results
The responses were sorted and separated into descriptive words or phrases due to
the fact that many panelists answered the question in a paragraph format. Exact wording
was preserved as much as possible. Duplicates or items with similar content were
combined and noted in a spreadsheet. Themes for each of the three open ended questions
were identified and items were grouped under the appropriate theme. There was a total of
173 distinct items identified by panelists overall in response to the three open ended
questions. Following, each of the three questions is described separately.
Research Question 1
RQ1 was “What are the indicators of digital literacy competency in teachers as
identified by a panel of experts?” This was addressed by panelists through DQ1, which
asked, “What qualities does a teacher possess and demonstrate that indicate a high
proficiency in digital and web literacy?” Responses to DQ1 were intended to create a
definition of a digitally literate teacher. This was posed as an open-ended question during
the first round of the current study.
Delphi Question 1
Five themes emerged from the panelist responses to DQ1. The themes were
characteristics, technology integration, digital and/or information literacy, pedagogy, and
miscellaneous. The 55 items were sorted by theme and alphabetized in preparation for
panelists to rate using a Likert-type scale in Round 2. Panelist language was preserved to
the fullest extent possible; the phrases listed were direct quotes.
Characteristics theme. The characteristics theme described personality traits that
could be ascribed to the definition of a digitally literate teacher. Items included in the
characteristics theme are included in Table 14.
Technology integration theme. The technology integration theme described how
a digitally literate teacher evaluated and used technology and integrated digital tools into
instruction. Items grouped under the technology integration theme are included in Table
15.
Table 14
Items Grouped Under Characteristics Theme for Open Ended Question 1
Ability and willingness to search for answers to their own questions
Ability to multi-task
Always-learning mentality
Ethics
Generosity in sharing
Growth mindset
Leadership in guiding teachers and students as they learn to navigate the Internet and use of
technological tools
Not afraid to try even if they’re pretty sure they’ll fail on the first attempt
Persistence when implementing technology; grit
Problem-solving skills
Risk-taking
Strong desire to put students first
Willingness to fail
Digital and/or information literacy theme. The digital and/or information
literacy theme described how a digitally literate teacher uses digital and information skills
Characteristics
in instruction. This included digital citizenship instruction. Items that were grouped under
the digital and/or information literacy theme are included in Table 16.
Table 15
Items Grouped Under Technology Integration Theme for Open Ended Question 1
Ability to explain the technology to others
Able/willing to troubleshoot issues as they arise
Affinity toward technology when addressing challenges
Comfortable with technology and willingness to explore
Embraces personal learning solutions through technology
Table 15 (continued)
Fluency with technology
Frequent use of technology and digital resources for preparing and delivering instruction
Amount of classroom content online and how often classroom instructional materials and
student content are stored online
Insatiable curiosity for new applications/curriculum tools which is then incorporated into
curriculum/training
Integration of technology in meaningful ways in the classroom and appropriately into
curriculum
Intuition about what to do when accessing a new website
Knowledge of many digital tools and resources
Positive attitude towards technology
Proper use of technology, especially in front of students
Self-empowered to experiment and learn about new web/digital literacies
Understanding of Substitution Augmentation Modification Redefinition (SAMR) model
Understanding that it's impossible to know everything there is to know about technology, and it
is acceptable to allow students to take the lead
Understanding that the technology is the tool to enhance learning
Understanding worlds of Apple, Microsoft, and Google as well as OER (Open Educational
Resources)
Uses digital tools to provide student and teacher choice
Use of technical tools efficiently and effectively to achieve personal and professional goals
Willingness to experiment with unfamiliar technologies
Technology Integration
Technology Integration
Pedagogy theme. The pedagogy theme contained items describing the ability of a
digitally literate teacher to appropriately apply pedagogy in a digital environment. Items
grouped under the pedagogy theme are included in Table 17.
Miscellaneous theme. The miscellaneous theme included items that did not match the
other four themes and were not in topics similar enough to form another theme.
Items that were grouped under the miscellaneous theme are included in Table 18.
Table 16
Items Grouped Under Digital and/or Information Literacy Theme for Open Ended
Question 1
Digital and/or Information Literacy
Ability to locate, evaluate, and use information effectively and efficiently
Ability to successfully instruct students in digital citizenship, digital and web literacy, research
skills, and copyright law
Digital literacy focus on high level, critical thinking ideas that aren't possible without the proper
use of technology
Skilled at being able to identify and synthesize information
Takes time to explain benefits and consequence of working online
Understanding and commitment to the principles of digital citizenship
Understanding of the importance of information literacy
Understanding of the importance of digital literacy
Table 17
Items Grouped Under Pedagogy Theme for Open Ended Question 1
Ability to take on different roles in the learning process with digital age learners e.g. facilitator,
learner, supporter, challenger
Achievement/licensure beyond school requirements
Knowledgeable about current strategies
Pedagogy
Knows how to develop curriculum based on Universal Design for Learning (UDL) model
Student centered, standards-aligned instruction
Understands pedagogy behind why using technology tools is the best for their learners
Of the items listed above, there were a number that were either exact duplicates or
contained content very similar in nature. Those items and the frequency of which they
were mentioned by the expert panelists are indicated in Table 19.
Table 18
Items Grouped Under Miscellaneous Theme for Open Ended Question 1
Miscellaneous
Awareness of International Society for Technology in Education (ISTE) and state standards
Eagerness to share/network new learning with peers at local, state, and national level
Membership in state, national, and/or international organizations
Understanding of equity and how that can lead to educational gaps with digital learning
Wagner’s 7 survival skills
Willingness to apply to present at local, state, and national conferences
Table 19
Items from Open Ended Question 1 mentioned by Two or More Panelists
Combined or Duplicate Items Frequency
Ability & willingness to search for answers to their own questions 2
Ability to successfully instruct students in digital citizenship, digital and web
literacy, research skills, and copyright law
5
Comfortable with technology and willingness to explore or experiment 6
Fluency with technology 2
Integration of technology in meaningful ways in the classroom and appropriately
into curriculum
4
Persistence when implementing technology; grit 2
Problem-solving skills 4
Risk-taking 4
Student centered, standards-aligned instruction 2
Understanding of Substitution Augmentation Modification Redefinition (SAMR)
model
3
Use of technical tools efficiently and effectively to achieve professional and
personal goals
2
Willingness to fail 4
Research Question 2
RQ2 was, “Which of these behaviors could be observed on a walk-through
observation?” RQ2 was addressed during the second round of the current study through
panelist answers to DQ2 and DQ3. Responses to the two Delphi questions are reported
separately.
Delphi Question 2
DQ2 was “What are the observable behaviors of a teacher with a high level of
digital literacy competence?” Four themes emerged from the panelist responses to DQ2.
The themes were characteristics, technology integration, digital and/or information
literacy, and pedagogy. The 57 items were sorted by theme and alphabetized in
preparation for panelists to rate using a Likert-type scale in subsequent rounds.
Participant language was preserved to the greatest extent possible and was corrected only
for spelling or grammatical errors. Exact wording was preserved so as to accurately
reflect panelist comments.
Teacher characteristics theme. The characteristics theme described personality
traits that could be ascribed to a teacher demonstrating a high level of digital literacy
competence in a classroom. Items included in the characteristics theme are included in
Table 20.
Teacher technology integration theme. The technology integration theme
encompassed items describing use of digital tools and how technology was integrated into
instruction by a teacher demonstrating a high degree of digital literacy competence in a
classroom. Items grouped under the technology integration theme are included in Table
21.
Table 20
Items Grouped Under Teacher Characteristics Theme for Open Ended Question 2
A “can-do” attitude
Constantly challenging themselves as learners, setting an example for their learners
Continuously seeking new knowledge
Lifelong learner
Reads, writes, shares and discovers daily
Strategic and intentional thinking
Willingness to take risks
Teacher digital and/or information literacy theme. The digital and/or
information literacy theme described how a teacher with a high level of digital literacy
competence used digital and information skills in classroom instruction. This included
digital citizenship instruction. Items that were grouped under the digital and/or
information literacy are included in Table 22.
Table 21
Items Grouped Under Teacher Technology Integration Theme for Open Ended Question 2
Ability to use a variety of technology resources
Able to help students use resources in engaging and meaningful ways for learning activities
Teacher Characteristics
Teacher Technology Integration
Analyzes technology so as to determine future societal impact
Attempts to understand the technology in order to determine its classroom application
Brings technology-related current events into the classroom when appropriate
Can sift through large amounts of information and chunk it easily
Demonstrates responsible use of technology to students
Demonstration of ease in learning and using new technologies
Differentiates content, process, and products using technology
Discerns between when using the technology tool is important vs. not needing to use it
Table 21 (continued)
Encourages students to use the technology most effective for their own learning rather than
assigning specific technology activities for the entire class
Ensures that students use the technology for learning more than he/she does for instruction
Expands the walls of the classroom
Explores new techniques through professional development
Frequent use of technology with classes, small groups, and individual students
Helps students to understand the impact of technology use on their lives (present & future)
Integration of digital tools and technologies or higher order thinking
Leadership role in encouraging use of technology by all members of the learning community
Models the use of technology for administrative tasks, and practices proper digital citizenship
both in the classroom and in personal use of technology
SAMR model for deeper learning
Students are exposed to varied tools (hardware and software) that support teaching and learning
Technology meaningfully integrated in the classroom that allows for creativity and productivity
Uses technology regularly and in a way that goes beyond substitution of SAMR model
Utilization of technology is completely integral to their teaching process
Willing to introduce new ideas or tools or technologies to students even when s/he may not yet
be an expert on them
Willingness to share experiences in use of technology and digital resources with colleagues
Table 22
Items Grouped Under Teacher Digital and/or Information Literacy Theme for Open
Ended Question 2
Teacher Digital and/or Information Literacy
Teacher Technology Integration
Actively uses social media and global communication tools, showing others about digital
citizenship
Cites their own sources and requires that of their students
Diligent about checking Terms of Service on digital resources and vets them before promoting
with students
Does not use online tools with terms of service that require consent or tools that do not protect
student/data/privacy
Table 22 (continued)
Teacher Digital and/or Information Literacy
Familiar with federal mandates, district Acceptable Use Policy, handbook, and other policies, to
ensure safe and effective use of digital resources
Finds teachable moments to model and remind students of the legal and ethical behaviors and
expectations associated with online access
Integrates digital literacies in lessons
Lessons are designed by the teacher to demonstrate accurate copyright information, digital
citizenship, and research skills
Licensed above required school certification to improve digital literacy
Proficient in online communication skills, computational thinking skills and digital citizenship
Publishes digital literacy efforts through text, video, podcast, etc.
Works with parents explaining legal and ethical laws and expectations associated with working
in a digital learning environment
Pedagogy theme. The pedagogy theme contained items describing the ability of a
teacher with a high level of competence in digital literacy to appropriately apply the
application of pedagogy in a digital environment. Items grouped under the pedagogy
theme are included in Table 23.
Table 23
Items Grouped Under Pedagogy Theme for Open Ended Question 2
Able to teach with multiple means of engagement for various developmental lessons as well as
design high quality assessments to check learning
Pedagogy
Acts as facilitator, providing students access to tools and resources in a manner that lets the
students be the experts
Cross-curricular education components
Disaggregates data to determine effectiveness of instruction
Innovative pedagogies
Lesson design allows students to learn from mistakes, choose the tools they will use, and have
the opportunity to design how they will synthesize their learning
Table 23 (continued)
Open-ended question to start the discussion
Presents at local, state, and national conferences
Provides tiered, personalized instruction with plenty of scaffolds using technology as a support
Shares with colleagues and invites them into the classroom
The teacher-student relationship is one of "meddler in the middle," not sage on the stage or
guide on the side
Universal Design for Learning (UDL) lesson plans
Of the items listed in Table 23, there were a number that were either exact
duplicates or contained content very similar in nature. Those items and the frequency of
which they were mentioned by the expert panelists are indicated in Table 24.
Table 24
Items from Open Ended Question 2 mentioned by Two or More Panelists
Combined or Duplicate Items Frequency
Differentiation 2
Digital citizenship instruction 3
Leadership 3
Models technology use 2
Offers student choice 5
Understands Terms of Service 3
Uses a variety of technology resources 3
Uses data 2
Willing to introduce new ideas or tools 3
Pedagogy
Willing to share ideas or collaborate 4
Delphi Question 3
DQ3 was “What are the observable behaviors of students in a classroom with a
teacher demonstrating a high level of digital literacy competence?” Four themes emerged
from the panelist responses to DQ3. The themes were characteristics, technology
integration, digital and/or information literacy, and instructional activities.
The 61 items were sorted by theme and alphabetized in preparation for panelists to rate
using a Likert-type scale during subsequent rounds. Participant language was preserved
to the greatest extent possible and was corrected only for spelling or grammatical errors.
Exact wording was preserved so as to accurately reflect panelist comments.
Student characteristics theme. The characteristics theme described personality
traits that could be ascribed to students taught in a classroom by a teacher demonstrating
a high level of digital literacy competence. Items included in the characteristics theme are
included in Table 25.
Table 25
Items Grouped Under Student Characteristics Theme for Open Ended Question 3
Collaborate to solve problems
Effective problem-solver
Empowered learners who ask and answer questions
Engaged learners
Entrepreneurial spirit where students feel empowered to follow their ideas and passions
Experimentation with new ideas
Have a lower fear of failure and a greater risk-taking attitude
Open to inquiry-based learning
Persistence
Self-sufficiency
Student Characteristics
Take risks, model, and try new things
Willing to take the risk of attempting something at which they may fail
Student technology integration theme. The technology integration theme
encompassed items describing student use of digital tools and how technology was
integrated into instruction in a classroom with a teacher demonstrating a high degree of
digital literacy competence in a classroom. Items grouped under the technology
integration theme are included in Table 26.
Table 26
Items Grouped Under Student Technology Integration Theme for Open Ended Question 3
Ability to find appropriate resources for own questions
Ability to troubleshoot when needed
Able to use digital tools with ease
Accessing what teachers have provided for them using the technologies available
Adept with and can teach others how to use tools
Apply their knowledge and ideas with technology
Begin to have an awareness of the scope of what is in their hands
Create products to demonstrate their learning
Curate resources for themselves and others
Demonstrate understanding when digital resources are best and when print resources are more
appropriate
Effectively use technology to create, organize, and access safely curated materials
Effortlessly self-select and evaluate the best tool(s) for the job at hand
Empowered by the technology to be self-actualized learners
Enthusiasm for learning how to use new technologies
Multiple ways both digital and non-digital for students to share their knowledge
Opportunities to speak, listen, read, write, and present with technology
Proficient in online research
Self-directing paths of locating information
Storing, using, and creating content in the classroom and storing online
Understand greater implications and evidence that are in their work
Technology Integration
Use a variety of digital tools appropriate for the product they are going to create
Use technology appropriately and effectively
Willingness to experiment with new technologies
Work collaboratively with their teacher, peers, and stakeholders outside their
classroom/building
Student digital and/or information literacy theme. The digital and/or
information literacy theme described how students used digital and information literacy
tools in a classroom taught by a teacher with a high level of digital literacy competence.
This included digital citizenship skills. Items that were grouped under the digital and/or
information literacy theme are included in Table 27.
Table 27
Items Grouped Under Student Digital and/or Information Literacy Theme for Open
Ended Question 3
Student Digital and/or Information Literacy
Comply with district policies such as the Acceptable Use Policy
Created work includes appropriate citation of sources
Demonstrate and practice all facets of digital citizenship at all times
Demonstrate appropriate online behavior, understanding the different types of plagiarism and
actively work to create their own materials
Demonstrate respect for intellectual property and appropriately credit others for their work
Grasp the concept of copyright and wrongs appropriate for their grade level
Show a consideration for privacy
Work with peers, teachers, and parents to obtain digital skills
Instructional activities theme. The instructional activities theme contained items
describing the learning-related actions of students in a classroom taught by a teacher with
a high level of competence in digital literacy. Items grouped under the instructional
activities theme are included in Table 28.
Of the items listed above, there were a number that were either exact duplicates or
contained content very similar in nature. Those items and the frequency of which they
were mentioned by the expert panelists are indicated in Table 29.
Table 28
Items Grouped Under Instructional Activities Theme for Open Ended Question 3
Able to communicate what they are doing
Ask each other for help rather than relying on the teacher
Engage in activities that require high levels of thinking and encourage collaboration
Evaluation
Have choice in how they do their work as well as who they work with
High level of creative expression in a judgment-free zone of learning
Lessons should involve a variety of modalities
Questioning
Reflection and repetition
Scaffolded autonomy
Shared discussion
Student-centered learning with outcomes that use critical thinking and attempting to solve
realworld problems
Technology isn't just an add-on or something they would learn in a silo and is instead woven
into lessons to build and grow student understanding
Understand different registers/purposes for digital communication
Understanding of design cycle/process
Working on clear academic objectives and behavioral norms set by the teacher and the
technology being used isn't the goal in itself
Write for a global audience
Round 1 Summary
Round 1 of the current study began on December 10, 2017 with an email sent to
38 experts who had previously indicated an interest in participating in the current study.
The email contained a link to an online survey instrument including the Informed
Instructional Activities
Consent document, basic demographic questions, and three open ended questions.
Table 29
Items from Open Ended Question 3 mentioned by Two or More Panelists
Combined or Duplicate Items Frequency
Able to use digital tools appropriately and effectively 8
Apply knowledge through technology 4
Demonstrate digital citizenship 4
Empowered learners 5
Engaged 7
Student-centered learning 2
Work collaboratively 6
Delphi Question 1 (DQ1) What qualities does a teacher possess and demonstrate that
indicate a high proficiency in digital and web literacy?
Delphi Question 2 (DQ2) What are the observable behaviors of a teacher with a high
level of digital literacy competence?
Delphi Question 3 (DQ3) What are the observable behaviors of students in a classroom
with a teacher demonstrating a high level of digital literacy competence? Round 1
remained open for panelist responses through January 3, 2018. At the conclusion of
Round 1, there was a 68% response rate with 26 of the original 38 volunteers
completing the three open ended questions. Expert panelists identified 173 unique
elements at the close of Round 1. These items are grouped by theme as demonstrated in
Table 30.
Delphi Round 2
The second round of the current study presented expert panelists with a survey
instrument created from the responses received in the first round and a summary of
responses from all panelists as shown in Appendices B and C. The survey included a list
of observable behaviors obtained from the first round that panelists were asked to rank
using a Likert-type scale of 1-5.
Table 30
Number of Items Grouped by Theme from Round 1 Responses
Question Theme Number of Items
Delphi Question 1
Characteristics 13
Technology Integration 22
Digital and/or Information Literacy 8
Pedagogy 6
Miscellaneous 6
Total for Question 1 55
Delphi Question 2
Teacher Characteristics 7
Teacher Technology Integration 26
Teacher Digital and/or Information Literacy 12
Pedagogy 12
Total for Question 2 57
Delphi Question 3
Student Characteristics 12
Student Technology Integration 24
Student Digital and/or Information Literacy 8
Instructional Activities 17
Total for Question 3 61
The Likert-type scale differed slightly for DQ1 than for DQ 2 and DQ3. RQ1 was
designed to determine elements to be included in the definition of a digitally literate
teacher. The rating scale for DQ1 was as follows:
1 = Should not be included in the definition
2 = Unimportant for use in the definition
3 = Depends on the circumstances if the current item should be included in the
definition
4 = Important to include in the definition
5 = Essential to include in the definition
DQ2 and DG3 were intended to gather a list of observable classroom behaviors in
order to create a walk-through form to determine the digital literacy proficiency level for
a teacher. The Likert-type scale used for DQ2 and DQ3 was as follows:
1 = Should not be included in the walk-through document.
2 = Unimportant for use in the walk-through document.
3 = Depends on the circumstances if the current item should be included in the
walk-through document.
4 = Important to include in the walk-through document.
5 = Essential to include in the walk-through document.
Timeline for Round 2
Expert panelists participating in Round 1 were sent an email with a link on
Surveymonkey.com containing the 173 items identified in Round 1. They were asked to
rate these items for suitability in a classroom walk-through document using two criteria:
the item pertains to digital and/or internet literacy and the item is observable during a
classroom walk-through. A Likert-type scale was provided as the measurement tool.
An email with a link to the survey was sent to panelists on January 15, 2018. A reminder
email was sent to panelists who had not completed the survey as of January 23, 2018. A
thank you email was sent to all panelists completing Round 2 of the current study after
the close of the round, which was January 26, 2018. Of the 26 original expert panelists
receiving the Round 2 invitation, 19 panelists completed Round 2.
Round 2 Analysis and Results
Responses at the close of Round 2 were entered into IBM SPSS software. Central
tendency statistical analysis for mean, median, mode, and standard deviation was
calculated to determine consensus for each item. For the purposes of the current study,
consensus on an item met one or more of the following criteria in order to be included in
the definition or in the walk-through document:
•70% or greater panel agreement;
•4.0 or greater mean score;
•4.0 or greater median and mode score; or
•1.0 or less standard deviation score.
Consensus on an item met one or more of the following criteria in order to be
eliminated from the definition or from the walk-through document:
•70% or greater panel agreement;
•2.99 or less mean score;
•2.99 or less median and mode score; or
•1.0 or less standard deviation score.
A Cronbach’s alpha statistical analysis of reliability and internal consistency was
also conducted on the 173 items at the conclusion of Round 2. Cronbach’s alpha showed
the questionnaire to reach acceptable reliability, α = 0.988. “The closer Cronbach’s alpha
coefficient is to 1.0 the greater the internal consistency of the items in the scale” (Gliem
& Gliem, 2003, p. 87).
Delphi Question 1
Responses to DQ1 were intended to create a definition of a digitally literate
teacher. Five themes emerged from the panelist responses to DQ1 during the first round
of the current study. These themes were characteristics, technology integration, digital
and/or information literacy, pedagogy, and miscellaneous. The 55 items identified during
Round 1 were sorted by theme and alphabetized in preparation for panelists to rate using
a Likert-type scale during Round 1.
Characteristics theme. The characteristics theme described personality traits that
could be ascribed to the definition of a digitally literate teacher. Thirteen of the 55 items
identified by panelists during Round 1 were grouped into the characteristics theme by the
researcher at the conclusion of the first round. These 13 items were presented to panelists
to rank with a Likert-type scale in Round 2. Of those, six items reached consensus for
inclusion in the definition of a digitally literate teacher. The items reaching consensus for
inclusion in the definition are described in Table 31.
Table 31
Items from Characteristics Theme Reaching Consensus for Inclusion in Definition During
Round 2
Item
number Description Mean Median Mode SD
1 Ability and willingness to search for
answers to their own questions
3.84 4 5 1.068
7 Leadership in guiding teachers and students
as they learn to navigate the Internet and use
of technological tools
4.05 4 5 0.970
9 Persistence when implementing
technology; grit
3.42 4 4 0.961
10 Problem-solving skills 3.84 4 4 1.015
11 Risk-taking 3.42 4 4 0.902
13 Strong desire to put students first 3.37 4 4 1.165
Despite reaching consensus for inclusion in the definition, expert panelists
demonstrated some conflicting opinions on several of the items. The comments referred
mainly to the two points panelists were asked to consider when rating each item: the item
pertains to digital and/or internet literacy, and the item is observable during a classroom
walk-through. Comments for items reaching consensus in the definition are included in
Table 32.
Table 32
Comments for Items from Characteristics Theme Reaching Consensus for Inclusion in
Definition During Round 2
Item
number Description Comment(s)
1
Ability and willingness to
search for answers to their
own questions
Behavior can be demonstrated, but is not limited to
Digital and web literacy
This may be difficult to observe on a walk-through,
but is important to the definition
Digital and web literacy includes this skill or
attribute
Teachers need to model this
7 Leadership in guiding teachers
and students as they learn to
navigate the Internet and use
of technological tools
None
9
Persistence when
implementing
technology; grit
I think this is true, but likely difficult to demonstrate
on a walk-through
Persistence based in belief, this is beneficial
10
Problem-solving skills
Not limited to digital literacy
This is an essential element of digital/web literacy
11
Risk-taking
Not limited to digital literacy
Same as a growth mindset
13 Strong desire to put students
first
I think this is something that a general educator
should do. It’s not specific to a technology teacher
Of the original 13 items, two reached consensus to be eliminated from the
definition of a digitally literate teacher. The ability to multitask (mean = 2.11, median = 2,
mode = 2, SD = 0.994) was deemed inappropriate by expert panelists according to one
panelist because
Research also shows that multitasking, i.e. trying to do two cognitive things at the
same time, simply can’t be done—the mind doesn’t work that way. Even trying to
parallel path a cognitive activity and a more automatic activity doesn’t really
work. That’s why the National Transportation Safety Board reports that texting
while driving is the equivalent of driving with a blood-alcohol level three times
the legal limit.
Another panelist commented, “We all do it but are the things we’re doing simultaneously
being done as well?”
The second item eliminated was “Generosity in sharing” (mean = 2.58, median =
3, mode = 3, SD = 0.961). Comments from panelists explained why this item was
eliminated. Panelists stated, “sharing is [too] broad” and “there is something to be said
about over-sharing.”
There were five items from the characteristics theme failing to reach consensus
for inclusion in the definition of a digitally literate teacher. These items are found in Table
33. Panelists demonstrated their lack of consensus on these items in the comments fields.
For example, the “Always-learning mentality” was deemed to be required and essential
but also to be not observable. Comments for all of the items in the characteristics theme
that failed to reach consensus are described in Table 34. These items were reevaluated by
panelists in Round 3.
Table 33
Items from Characteristics Theme Failing to Reach Consensus for Definition During
Round 2
Item
number Description Mean Median Mode SD
3 Always-learning mentality 3.05 3 3 1.433
4 Ethics 3.79 4 3* 1.032
6 Growth mindset 3.47 3 3 1.264
8 Not afraid to try even if they’re pretty sure
they’ll fail on the first attempt
3.16 3 3* 1.167
12 Strong desire to put students first 3.32 3 3 1.336
Note: Multiple modes exist for Items 4 and 8 as reported by SPSS. The smallest value is
shown.
Table 34
Comments for Items from Characteristics Theme Failing to Reach Consensus for
Definition During Round 2
Item
number Description Comments
3
Always-learning mentality
Technology is a field that requires this mentality
Not quantifiable and observable
Useful to the definition, hard to observe in a
walkthrough
Lifelong learner and ability to find answers to
information problems = essential
Same as growth mindset
4
Ethics
I need more than just a word. What is ethical? I
think it should be is it legal?
We must model ethical behavior if we expect it from
our students
Again, important to model for students.
6
Growth mindset
I think “always learning” better encapsulates this
than “growth mindset”
The idea that learning is not finite is an important
element of a digital/web literate person This
term is so over used at this point.
8
Not afraid to try even if
they’re pretty sure they’ll fail
on the first attempt
Risk-taking important for teachers; we can no
longer be the experts on everything we are expected
to teach
This is the same as a growth mindset
12 Strong desire to put students
first
I think this is something that a general educator
should do. It’s not specific to a technology teacher
Technology integration theme. The technology integration theme described how
a digitally literate teacher evaluated and used technology and integrated digital tools into
instruction. Twenty-two of the 55 items identified by panelists during Round 1 were
grouped into the technology integration theme by the researcher at the conclusion of the
first round. Of those, there were 15 items achieving consensus for inclusion in the
definition of a digitally literate teacher. They are listed in Table 35. While comments on
these items were favorable for the most part, there were some panelists who felt these
items were difficult to observe. All comments for the items reaching consensus for
inclusion are listed in Table 36.
Table 35
Items from Technology Integration Theme Reaching Consensus for Inclusion in Definition
During Round 2
Item
number Description Mean Median Mode SD
14 Ability to explain the technology to others 4.16 4 4 0.834
15 Able/willing to troubleshoot issues as they arise 3.95 4 4 0.705
16 Affinity toward technology when addressing
challenges
3.47 4 4 0.905
17 Comfortable with technology and willingness to
explore
4.05 4 4 0.970
18 Embraces personalized learning solutions
through technology
3.78 4 4 0.943
19 Fluency with technology 4.11 4 4 0.832
20 Frequent use of technology and digital resources
for preparing and delivering instruction
3.89 4 3 0.900
23 Integration of technology in meaningful ways in
the classroom and appropriately into curriculum
4.47 5 5 0.772
26 Positive attitude towards technology 3.89 4 4* 1.023
28 Self-empowered to experiment and learn about
new web/digital literacies
4.00 4 4* 1.054
29 Understanding that it's impossible to know
everything there is to know about technology,
and it is acceptable to allow students to take the
lead
4.00 4 4* 1.291
Table 35 (continued)
Item Description Mean Median Mode SD
number
30 Understanding that the technology is the tool to
enhance learning
4.05 4 5 0.911
33 Uses digital tools to provide student and teacher
choice
3.95 4 4 0.911
34 Use of technical tools efficiently and
effectively to achieve professional and personal
goals
3.79 4 4 0.958
35 Willingness to experiment with unfamiliar
technologies
3.84 4 4 0.958
Note: Multiple modes exist for items 26, 28, and 29 as reported by SPSS. The smallest
value is shown.
Table 36
Comments for Items from Technology Integration Theme Reaching Consensus for
Inclusion in Definition During Round 2
Item
number Description Comments
14
Ability to explain the technology
to others
This behavior is easily observable and can be
documented in your study. The depth of
understanding required to not only transfer the
information understood in a student's personal
situation, but having the necessary skills to translate
that knowledge to apply it in new situations as they
help others (different application/different types of
devices/different operating systems, different
learning platforms/different programs or apps) helps
the learner synthesize and deepen their learning
Sometimes being an expert on something doesn’t
translate to being a good teacher of that something
Need to be able to communicate information If you
cannot explain how to use it and what it is then how
can you be literate?
15
Able/willing to troubleshoot issues
as they arise
Yes and no. I think there’s literate and there’s
expertise. One can be literate in a domain without
being an expert.
Might not work but IT isn’t always there, right?
technical skills are necessary Depends
on tech support available
Table 36 (continued)
Item
number Description Comments
16
Affinity toward technology when Affinity - not observable addressing
challenges
Technology should be used when it makes sense or
improves the learning but not necessarily the “go
to”
If “affinity toward” is regarded as similar to
“essential to consider including”
17 Comfortable with technology and I like fluent with tech better- more precise and a
willingness to explore term that is measurable
18
Embraces personalized learning
solutions through technology
Believe this to be very important but not as essential
for my definition of digital/web literacy. I see
Personalized Learning as the next phase once staff
are highly digitally/web literate.
I like this, but I don’t think this is required to
demonstrate literacy
Embraces vs uses? Is embracing observable? Not
sure of the word embraces
19 Fluency with technology I think willingness to try and fail leads to fluency
but the person with the right dispositions might not
start out that way
20
Frequent use of technology and
digital re-sources for preparing
and delivering instruction
Frequent use is not as important as effective use
Frequent might be “appropriate” instead
Depends on whether technology supports/enhances
the learning
Again, the key is using it when it makes sense to the
learning
I think this is important but really depends on
content being taught
Isn’t this implied already?
23 Integration of technology in This statement needs to be extended: “meaningful
meaningful ways in the classroom ways that leads to creation or presenting
and appropriately into curriculum information and understanding in ways not possible
without technology
26 Positive attitude towards
technology
Not quantifiable and observable
28
Self-empowered to experiment and
learn about new web/digital
literacies
I don’t like this one because it’s vague. Can I
measure this?
I like “learn about new web/digital literacies”
29 Understanding that it’s impossible
to know everything there is to
know about technology, and it is
acceptable to allow students to
take the lead
Again, is this specific to technology? Any good
teacher should be allowing students to take the lead.
Table 36 (continued)
Item
number Description Comments
30 Understanding that the
technology is the tool to enhance
learning
It’s “a” tool, not necessarily “the” tool.
33 Uses digital tools to provide
student and teacher choice
None
34 Use of technical tools efficiently
and effectively to achieve
professional and personal goals
None
35
Willingness to experiment with
unfamiliar technologies
I like this, but worry about ability to measure on a
walk-through
Willingness—just experiments with unfamiliar
technologies
Of the original 22 items grouped in the technology integration theme presented to
the expert panelists, three of those items reached consensus to be eliminated from the
definition of a digitally literate teacher. They are described in Table 37. Panelists were in
agreement about eliminating these items according to their comments. The items were
described as unclear and not observable. All comments for eliminated items in the
technology integration theme are included in Table 38.
Table 37
Items from Technology Integration Theme Reaching Consensus for Elimination from
Definition During Round 2
Item
number Description Mean Median Mode SD
21 How much classroom content is online and how
often classroom instructional materials and
student content stored online
2.79 3 2* 1.182
24 Intuition about what to do when accessing a new
website
2.79 3 3 1.316
32 Understands worlds of Apple, Microsoft, and
Google as well as Open Educational Resources
(OER)
2.79 3 3 1.273
Note: Multiple modes exist for Item 21 as reported by SPSS. The smallest value is shown.
Table 38
Comments for Items from the Technology Integration Theme Reaching Consensus for
Elimination from Definition During Round 2
Item
number Description Comments
21
How much classroom content
is online and how often
classroom instructional
materials and student content
stored online
The tech is a tool to enhance learning so the person
should be able to fluidly decide when to use it and
when not to.
Unclear what this means.
24 Intuition about what to do
when accessing a new website
None
32
Understands worlds of
Apple, Microsoft, and
Google as well as Open
Educational Resources (OER)
I like this question, but not the phrasing. I like that it
demonstrates the importance of knowing multiple
ecosystems. I don’t like it because the companies
listed may change over time. I’d be comfortable if
this was phrased as a “such as Apple, Microsoft,
etc.” question
Not sure what “worlds” means
Not quantifiable and observable-can you see this in a
walk through?
Could those worlds be considered finite? Won’t they
necessarily change/morph?
Of the 22 items in the technology integration theme, expert panelists failed to
reach consensus on four items. These were presented to the panelists in Round 3 for
reevaluation. The items moving on to Round 3 of the current study are listed in Table 39.
Comments from the panelists appear to demonstrate a lack of support for these
items despite the fact that they were not rated low enough to be eliminated from the
definition. Panelists particularly disfavored the item about the SAMR model of
technology integration. Comments for all of the items from the technology integration
theme are included in Table 40.
Table 39
Items from Technology Integration Theme Failing to Reach Consensus for Definition
During Round 2
Item
number Description Mean Median Mode SD
22 Insatiable curiosity for new
applications/curriculum tools which is then
incorporated into curriculum/training
3.05 3 4 1.129
25 Knowledge of many digital tools and resources 3.47 4 3* 1.219
27 Proper use of technology, especially in front of
students
3.58 3 3 1.071
49 Understanding of the Substitution
Augmentation Modification Redefinition
(SAMR) model
3.11 3 4 1.132
Note: Multiple modes exist for item 25 as reported by SPSS. The smallest value is shown.
Table 40
Comments for Items from the Technology Integration Theme Failing to Reach Consensus
for Definition During Round 2
Item
number Description Comments
22
Insatiable curiosity for new
applications/ curriculum tools
which is then incorporated
into curriculum/training
I object to the latter half . . . yes, look for new tools.
But the new tool is not always better.
Not in definition however I do feel ability to create is
important
25
Knowledge of many digital
tools and resources
This is the role of an instructional technologist, not a
literate teacher
Along with the willingness to explore new ones and
not adhere to the familiar
I am 100% digitally literate and my students know
more than I.
May be easily combined into other phrases already
considered in #1-24
27
Proper use of technology,
especially in front of students
I’m lukewarm on this one. I’m OK with students
seeing failure, as long as the teacher realizes that
they’re using technology improperly.
Specifically for copyright and digital citizenship We
can make mistakes. It is ok!
Table 40 (continued)
Item
number Description Comments
49
Understanding of the
Substitution Augmentation
Modification Redefinition
(SAMR) model
SAMR is a doctoral dissertation. While I think my
own doctoral dissertation is really swell, I’m not
sure that knowledge of my work is a precursor to
being an effective classroom teacher.
Not demonstrable
Is SAMR the best model? There is a bit of
controversy over this.
SAMR is NOT research based ...
Digital and/or information literacy theme. The digital and/or information
literacy theme described how a digitally literate teacher uses digital and information skills
in instruction, including digital citizenship instruction. Eight of the 55 items identified by
panelists during Round 1 were grouped into the digital and/or information literacy theme
by the researcher at the conclusion of the first round. Of those, six items reached
consensus during Round 2, as indicated in Table 41. Even though these items reached
consensus for inclusion, comments by panelists were not particularly favorable.
Comments for the items reaching consensus are included in Table 42.
One of the eight items in the digital and/or information literacy theme presented to
panelists in Round 2, “Takes time to explain benefits and consequences of working
online” (mean = 2.95, median = 3, mode - 3, SD = 1.079) reached consensus to be
eliminated from the definition of a digitally literate teacher.
One of the items, “Skilled at being able to identify and synthesize information”
(mean = 3.68, median = 4, mode = 3, SD = 1.157) failed to reach consensus. According to
panelist comments, the item was “not limited to digital literacy” and something a good
teacher would do. One panelist objected to the word “identify.” The item was included in
Round 3 for the expert panelists to review again.
Table 41
Items from Digital and/or Information Literacy Theme Reaching Consensus for Inclusion
in Definition During Round 2
Item
number Description Mean Median Mode SD
36 Ability to locate, evaluate, and use information
effectively and efficiently 4.26 5 5 0.872
37 Ability to successfully instruct students in digital
citizenship, digital and web literacy, research
skills, and copyright law
4.68 5 5 0.671
38 Digital literacy focus on high level, critical
thinking ideas that aren't possible without the
proper use technology
4.00 4 5 1.202
41 Understanding and commitment to the principles
of digital citizenship
3.89 4 5 1.286
42 Understanding of the importance of information
literacy
4.32 4 5 0.749
43 Understands the importance of digital literacy 4.11 4 5 0.994
Pedagogy theme. The pedagogy theme contained items describing the ability of a
digitally literate teacher to appropriately apply pedagogy in a digital environment. There
were six items grouped under the pedagogy theme presented to the expert panelists. Of
those items, two reached consensus to be included in the definition of a digitally literate
teacher.
1) “Understands pedagogy behind why using technology tools is the best for their
learners” (mean = 4.26, median = 5, mode = 5, SD = 0.872) reached consensus according
to one comment due to the fact that this element was “too often over-looked and/or
misunderstood.” 2)“The ability to take on different roles in the learning process with
digital age learners e.g. facilitator, learner, supporter, challenger” (mean = 4.16, median =
4, mode = 5, SD = 1.119) also reached consensus for inclusion.
Table 42
Comments for Items from Digital and/or Information Literacy Theme Reaching
Consensus for Inclusion in Definition During Round 2
Item
number Description Comments
36 Ability to locate, evaluate, and use information
effectively and efficiently
Not limited to digital resources
37 Ability to successfully instruct students in digital
citizenship, digital and web literacy, research
skills, and copyright law
Successfully is vague
38 Digital literacy focus on high level, critical
thinking ideas that aren't possible without the
proper use technology
I don’t know what this means.
41 Understanding and commitment to the principles
of digital citizenship
I’m not fond of “digital
citizenship.” I think it’s an
illdefined term.
42
Understanding of the importance of information
literacy
Not limited to digital literacy, not
demonstrable
I think all of these fit into a
broader category. You do not want
to be too specific in defining this.
43
Understands the importance of digital literacy
Not demonstrable
42 does this
Of the five items under the pedagogy theme, two of them reached consensus to be
eliminated from the definition of a digitally literate teacher. “Knows how to develop
curriculum based on the Universal Design for Learning (UDL) model” (mean = 2.74,
median = 3, mode = 3, SD = 1.046) was eliminated according to expert panelist
comments because it was “district specific in some cases.” The item was not pertinent to
the definition of a digitally literate teacher according to another panelist because “we’re
not evaluating a curriculum director… this is a classroom teacher.”
“Achievement/licensure beyond school requirements” (mean = 2.58, median = 3, mode = 2,
SD = 0.902) was eliminated according to one panelist, “if this is a walk-through instrument, I
don’t like this one because I can check the box every year for a certification that’s 10 years
old.”
Two items from the pedagogy theme failed to reach consensus in Round 2.
“Knowledgeable about current strategies” (mean = 3.44, median = 3.5, mode = 3, SD =
1.247) received the following comments from panelists:
•Not limited to digital literacy, not demonstrable
•Current strategies about what?
•I would like to have seen those outlined here before including it...
“Student centered, standards-aligned instruction” (mean = 3.44, median = 3.5, mode = 5,
SD = 1.423) was determined to be “something every teacher should do” and “not
actionable” by panelists providing comments on the item. Both of these items were
presented to expert panelists for reconsideration during Round 3.
Miscellaneous theme. The miscellaneous theme included items that did not
match the other four themes and were not in topics similar enough to form another theme.
Six items that were grouped under the miscellaneous theme were presented to the expert
panelists for evaluation using a Likert-type scale.
Two of the items reached consensus for inclusion in the definition of a digitally
literate teacher. The first item, “Awareness of International Society for Technology in
Education (ISTE) and state standards” (mean = 4.11, median = 5, mode = 5, SD = 1.150)
despite conflicting panelists comments:
•“Awareness doesn’t imply use”
•“Yes, knowing the national organization and its standards—whether you
believe them all or not—is important”
•“Pick a methodology like ISTE and go with it, rather than having several
options to cloud the understanding”
“Understanding of equity and how that can lead to educational gaps with digital
learning” (mean = 3.42, median = 4, mode = 5, SD = 1.502) reached consensus for
inclusion despite one panelist presenting a specific reason for not including the item in
the definition of a digitally literate teacher. The panelist wrote, “I was bounced from a job
interview in the first round because I was asked to define equity, and I placed it in a
digital context (FYI, my answer was right in the context of the interview). I think equity
is too broadly-defined (too many people have too many opinions) for it to be a good
metric.”
Of the six items presented to the expert panelists from the miscellaneous theme,
four of them reached consensus to be eliminated from the definition of a digitally literate
teacher. These items are described in Table 43.
Table 43
Items from Miscellaneous Theme Reaching Consensus for Elimination from Definition
During Round 2
Item
number Description Mean Median Mode SD
51 Eagerness to share/network new learning with
peers at a local, state, and national level
2.95 3 4 1.177
52 Membership in state, national, and/or
international organizations
2.53 2 2 1.020
53 Willingness to apply to present at local, state,
and national conferences
2.42 2 3 1.071
55 Wagner’s 7 survival skills 2.35 2 3 1.169
Panelists agreed in their comments these items were extraneous particularly due to
the items not lending themselves to observation or being too expensive for districts to
require. All comments about items eliminated from the definition in the miscellaneous
theme are listed in Table 44.
Table 44
Comments for Items from Miscellaneous Theme Reaching Consensus for Elimination
from Definition During Round 2
Item
number Description Comments
51
Eagerness to share/network
new learning with peers at a
local, state, and national level
Not necessarily observable in a walk-through
Eagerness--- not quantifiable and observable in a
walk through
Communication is important
52
Membership in state, national,
and/or international
organizations
Not all districts would provide the required funding.
Membership means less than participation,
contributions, etc.
Expensive if paying on their own....
Could be a monetary issue ...
53
Willingness to apply to present
at local, state, and
national conferences
Important, but not all districts would provide
funding
Some districts can’t do that... can you see it during a
walk-through?
This modeling has renewed meaning but should not
expect self-pay
55
Wagner’s 7 survival skills
I have two Ivy League degrees and a 20+ year
career working in technology in education, and I
don’t know what this is. Not actionable
Not familiar with these
I’m not familiar with this, thus a neutral answer
There is more than just him- What about DOK?
Delphi Question 2
Panelists were also asked during Round 2 to rate the 57 items for DQ2 identified
in Round 1. Four themes emerged from the 57 items: characteristics, technology
integration, digital and/or information literacy, and pedagogy. The 57 items were sorted
by theme and alphabetized in preparation for panelists to rate using a Likert-type scale
during Round 2. Participant language was preserved as much as possible when presenting
the 57 items from DQ2 for rating.
Teacher characteristics theme. The characteristics theme described personality
traits that could be ascribed to a teacher demonstrating a high level of digital literacy
competence in a classroom. A total of seven items of the 57 were grouped under the
characteristics theme to be rated by expert panelists. Of the seven items, only one,
“Constantly challenging themselves as learners, setting an example for their learners,”
(mean = 3.37, median = 4, mode = 4, SD = 1,422) reached consensus for inclusion in the
walk-through document.
Three of the seven items in the characteristics theme reached consensus for
elimination from the walk-through document. They are described in Table 45. Comments
for these items noted they were “not limited to digital literacy,” “not needed daily,” and
“not the responsibility of teachers.” Two of the items, “A ‘can-do’ attitude” and “Reads,
writes, shares, and discovers daily” were deemed to be elements all teachers should
possess.
Three of the seven items in the characteristics theme failed to reach consensus.
Those three items were included in Round 3 for reevaluation by the expert panelists. The
items are described in Table 46. Comments from panelists indicated all of these items
were something “All teachers should have.” In addition, “Continuously seeking new
knowledge” was deemed to be not observable.
Table 45
Items from Teacher Characteristics Theme Reaching Consensus for Elimination from
Document During Round 2
Item
number Description Mean Median Mode SD
56 A “can-do” attitude 2.89 3 2 1.487
60 Reads, writes, shares, and discovers daily 2.58 3 3 0.961
61 Strategic and intentional thinking 2.95 3 3 1.177
Table 46
Items from Teacher Characteristics Theme Failing to Reach Consensus for Document
During Round 2
Item
number Description Mean Median Mode SD
58 Continuously seeking new knowledge 3.21 3 5 1.512
59 Lifelong learner 3.32 3 5 1.376
62 Willingness to take risks 3.32 3 3* 1.108
Note: Multiple modes exist for item 62 as reported by SPSS. The smallest value is shown.
Teacher technology integration theme. Of the 57 total items presented to
panelists for DQ2, the technology integration theme encompassed 27 items. These items
described the use of digital tools and how technology was integrated into instruction by a
teacher demonstrating a high degree of digital literacy competence in a classroom. Of the
27 items, 19 of them reached consensus for inclusion in the non-evaluative classroom
walk-through document. These items are listed in Table 47. Despite these items reaching
consensus for inclusion, several of them received negative comments from participants.
All comments for the items reaching consensus for inclusion in the walk-through
document are listed in Table 48.
Of the 27 items in the technology integration theme, four reached consensus for
elimination from the walk-through document. They are described in Table 49.
Table 47
Items from Teacher Technology Integration Theme Reaching Consensus for Inclusion in
Document During Round 2
Item
number Description Mean Median Mode SD
63 Ability to use a variety of technology resources 3.89 4 4 0.963
64 Able to help students use resources in engaging
and meaningful ways for learning activities
3.83 4 4* 1.150
66 Attempts to understand the technology in order
to determine its classroom application
3.89 4 4 0.994
67 Brings technology-related current events into the
classroom when appropriate
3.11 3 3 0.937
69 Demonstrates responsible use of technology to
students
4.37 4 5 0.684
70 Demonstration of ease in learning and using new
technologies
3.26 4 4 1.195
71 Differentiates content, process, and products
using technology
3.68 4 4 1.057
72 Discerns between when using the technology
tool is important vs. not needing to use it
4.00 4 5 1.202
73 Encourages students to use the technology most
effective for their own learning rather than
assigning specific technology activities for the
entire class
4.00 4 5 1.106
77 Frequent use of technology with classes, small
groups, and individual students
3.50 3.5 3* 0.985
78 Helps students to understand the impact of
technology use on their lives (present & future)
3.78 4 4 1.060
79 Integration of digital tools and technologies for
higher order thinking
3.89 4 5 1.150
81 Models the use of technology for administrative
tasks and practices proper digital citizenship
both in the classroom and in personal use of
technology
3.95 4 5 1.079
82 Students are exposed to varied tools (hardware
and software) that support teaching and learning
3.74 4 4 0.933
Table 47 (continued)
Item
number Description Mean Median Mode SD
83 Technology is meaningfully integrated in the
classroom that allows for creativity and
productivity
4.53 5 5 0.697
84 Utilization of technology is completely integral
to their teaching process
3.79 4 4 0.976
85 Willing to introduce new ideas, tools or
technologies to students even when s/he may not
yet be an expert on them
4.00 4.5 5 1.138
112 Willingness to share experiences in use of
technology and digital resources with colleagues
3.32 3 3 0.946
Table 48
Comments for Items from Teacher Technology Integration Theme Reaching Consensus for
Inclusion in Document During Round 2
Item
number Description Comments
63
Ability to use a variety of technology resources
Ability is not the same as skill
Depends what you mean by “use”
64
Able to help students use resources in engaging
and meaningful ways for learning activities
Not limited to digital resources
All teachers should have this
“Facilitate” would be a better
word than “help”
66 Attempts to understand the technology in order to
determine its classroom application
Not observable
70 Demonstration of ease in learning and using new
technologies
I don’t understand this
71 Differentiates content, process, and products
using technology
None
72
Discerns between when using the technology
tool is important vs. not needing to use it
Afraid that this could be an excuse
to NOT use technology Not
observable
73
Encourages students to use the technology most
effective for their own learning rather than
assigning specific technology activities for the
entire class
No, there are times when all
students need to know the tool . . .
even if they’re not going to
personally use it
Individualized instruction based
on technology resource
Table 48 (continued)
Item
number Description Comments
77 Frequent use of technology with classes, small
groups, and individual students
Frequency is not synonymous
with effectiveness
78 Helps students to understand the impact of
technology use on their lives (present & future)
None
79 Integration of digital tools and technologies for
higher order thinking
None
81 Models the use of technology for administrative
tasks and practices proper digital citizenship both
in the classroom and in personal use of technology
None
82 Students are exposed to varied tools (hardware and
software) that support teaching and learning
May not be an option
83 Technology is meaningfully integrated in the
classroom that allows for creativity and
None
productivity
84 Utilization of technology is completely integral to
their teaching process
None
85 Willing to introduce new ideas, tools or
technologies to students even when s/he may not
yet be an expert on them
None
112 Willingness to share experiences in use of
technology and digital resources with colleagues
None
Table 49
Items from Teacher Technology Integration Theme Reaching Consensus for Elimination
from Document During Round 2
Item
number Description Mean Median Mode SD
65 Analyzes technology so as to determine future
societal impact
2.89 3 3 0.937
68 Can sift through large amounts of information
and chunk it easily
2.53 3 3 1.020
76 Explores new techniques through professional
development
2.84 3 3 1.015
86 SAMR model for deeper learning 2.78 3 3 1.114
Panelists were particularly unfavorable about the item pertaining to the SAMR
model of technology integration. Despite the wide use of this model in classrooms and as
a topic for professional development, it was not deemed necessary for inclusion in the
walk-through document due to the following reasons noted in the comments field for this
item:
•“Not actionable”
•“Again, this is a doctoral dissertation. While my own dissertation is absolutely
BEDAZZLING, knowledge of mine is not required to be a good teacher with
technology”
•“Again, I question the validity of the SAMR model. Perhaps TIMS is better?”
•“SAMR is not research based”
Comments for the other items eliminated from the walk-through document were
listed in Table 50.
Table 50
Comments for Items from Teacher Technology Integration Theme Reaching Consensus for
Elimination from Document During Round 2
Item
number Description Comments
65 Analyzes technology so as to
determine future societal
impact
Yikes! I think this is an expectation for a futurist.
It’s a lot to ask for a teacher.
68 Can sift through large amounts
of information and chunk it
easily
76
Explores new techniques
through professional
development
What techniques? Technology ones?
PD is not a statistically valid strategy according to
Hattie
Four of the 27 items in the technology integration theme failed to reach
consensus. Those four items were included in Round 3 for reevaluation by the expert
panelists. The items are described in Table 51. Comments from expert panelists describe
the items failing to reach consensus as vague, unnecessary, or difficult to observe. All
comments for these items are listed in Table 52.
Table 51
Items from Teacher Technology Integration Theme Failing to Reach Consensus During
Round 2
Item
number Description Mean Median Mode SD
74 Ensures that students use the technology for
learning more than he/she does for instruction
3.21 3 4 1.032
75 Expands the walls of the classroom 3.37 3 3 1.257
80 Leadership role in encouraging use of
technology by all members of the learning
community
3.21 3 4 1.084
87 Uses technology regularly and in a way that
goes beyond substitution level of SAMR model
3.16 3 4 1.259
Teacher digital and/or information literature literacy theme. The digital
and/or information literacy theme described how a teacher with a high level of digital
literacy competence uses digital and information skills in classroom instruction. This
could include digital citizenship instruction. Of the 57 items total for DQ2, 12 items were
grouped under the digital and/or information literacy theme and presented to expert
panelists for rating using a Likert-type scale. Of the 12 items, nine reached consensus for
inclusion in the classroom walk-through document, as described in Table 53. Despite
reaching consensus for inclusion in the walk-through document, several of the items
received negative comments from the expert panelists. All comments for these items are
listed in Table 54.
Table 52
Comments for Items from Teacher Technology Integration Theme Failing to Reach
Consensus During Round 2
Item
number Description Comments
74
Ensures that students use the technology for
learning more than he/she does for instruction
I’m OK with both
Difficult to observe
75 Expands the walls of the classroom The teacher’s calling a contractor?
I know what the author means . . .
this is just way too vague.
80 Leadership role in encouraging use of
technology by all members of the learning
community
No. I don’t think being a facile
teacher requires that you exhibit
leadership in it.
87 Uses technology regularly and in a way that
goes beyond substitution level of SAMR
model
Again, this is a doctoral
dissertation. While my own
dissertation is absolutely
BEDAZZLING, knowledge of
mine is not required to be a good
teacher with technology.
Again, I question the validity of the
SAMR model. Perhaps TIMS is
better?
Table 53
Items from Teacher Technology Integration Theme Reaching Consensus for Inclusion in
Document During Round 2
Item
number Description Mean Median Mode SD
98 Actively uses social media and global
communication tools, showing others about
digital citizenship
3.32 4 4 1.250
99 Cites their own sources and requires that of their
students
3.68 4 5 1.250
100 Diligent about checking Terms of Service on
digital resources and vets them before
promoting with students
3.74 4 5 1.447
101 Does not use online tools with terms of service
that require consent or tools that do not protect
student/data/privacy
3.58 4 5 1.465
Table 53 (continued)
Item
number Description Mean Median Mode SD
102 Familiar with federal mandates, district
Acceptable Use Policy (AUP), handbook, and
other policies, to ensure safe and effective use of
digital resources
4.33 4.5 5 0.840
103 Finds teachable moments to model and remind
students of the legal and ethical behaviors and
expectations associated with online access
4.16 4 4 0.765
104 Integrates digital literacies in lessons 4.32 4 5 0.749
105 Lessons are designed by the teacher to
demonstrate accurate copyright information,
digital citizenship, and research skills
3.95 4 4 0.911
108 Proficient in online communication skills,
computational thinking skills and digital
citizenship
4.06 4 4 0.802
Table 54
Items from Teacher Technology Integration Theme Reaching Consensus for Inclusion in
Document During Round 2
Item
number Description Comments
98 Actively uses social media and global
communication tools, showing others about
digital citizenship
This is a personal preference
99 Cites their own sources and requires that of their
students
None
100
Diligent about checking Terms of Service on
digital resources and vets them before
promoting with students
I think this is more the role of
Director of Technology/CIO
Not observable
101
Does not use online tools with terms of service
that require consent or tools that do not protect
student/data/privacy
Again, I think others in the
district are responsible for vetting.
Observable only to the extent that
the observer knows the terms and
service of the technology being
used
They could use them for
themselves, this question in not
clearly enough written. Should
say students.
Table 54 (continued)
Item
number Description Comments
102 Familiar with federal mandates, district
Acceptable Use Policy (AUP), handbook, and
other policies, to ensure safe and effective use of
digital resources
Most teachers are not familiar so
maybe putting it out there will
help.
103 Finds teachable moments to model and remind
students of the legal and ethical behaviors and
expectations associated with online access
None
104 Integrates digital literacies in lessons None
105 Lessons are designed by the teacher to
demonstrate accurate copyright information,
digital citizenship, and research skills
None
108 Proficient in online communication skills,
computational thinking skills and digital
citizenship
Not quantifiable and observable
Of the 12 items in the digital and/or information literacy theme, one item reached
consensus for elimination from the walk-through document and one item failed to reach
consensus. “Publishes digital literacy efforts through text, video, podcast, etc.” (mean =
2.84, median =3=, mode =3=, SD = 1.167) was eliminated from the document. “Works
with parents explaining the legal and ethical laws and expectations associated with
working in a digital learning environment” (mean = 3.32, median = 3, mode = 2, SD =
1.250) failed to reach consensus. The lone comment from an expert panelist stated a
possible reason, “I almost feel like this is a job of an instructional technologist, not a
teacher.” This item was included in Round 3 for reevaluation by panelists.
Pedagogy theme. Of the 57 total items presented to panelists for DQ2, the
pedagogy theme encompassed 12 items. The pedagogy theme contained items describing
the ability of a teacher with a high level of competence in digital literacy to appropriately
apply the application of pedagogy in a digital environment. These were presented to
expert panelists for rating using a Likert-type scale. Of the 12 items, four reached
consensus for inclusion in the classroom walk-through document, as described in Table
55.
Panelist comments about these items were centered primarily around “Able to
teach with multiple means of engagement for various developmental lessons as well as
design high quality assessments to check learning” in particular. This item was “not
limited to digital resources” and did not mention technology. The other comment was in
response to “Provides tiered, personalized instruction with plenty of scaffolds using
technology as a support” as being an item that was “unclear.”
Table 55
Items from Pedagogy Theme Reaching Consensus for Inclusion in Document During
Round 2
Item
number Description Mean Median Mode SD
88 Able to teach with multiple means of
engagement for various developmental lessons as
well as design high quality assessments to check
learning
3.56 4 4 1.247
89 Acts as facilitator, providing students access to
tools and resources in a manner that lets the
students be the experts
3.89 4 4 0.875
93 Lesson design allows students to learn from
mistakes, choose the tools they will use, and
have the opportunity to design how they will
synthesize their learning
3.95 4 4 0.780
95 Provides tiered, personalized instruction with
plenty of scaffolds using technology as a support
3.63 4 4 1.012
Of the 12 items in the pedagogy theme, five reached consensus for elimination
from the walk-through document. They are described in Table 56. Comments from the
panelists noted the items were not necessary or not actionable. The complete list of
comments regarding these items was included in Table 57.
Table 56
Items from Pedagogy Theme Reaching Consensus for Elimination from Document
During Round 2
Item
number Description Mean Median Mode SD
90 Cross-curricular education components 2.78 3 3 0.943
94 Open-ended question to start the discussion 2.83 3 2 1.20
96 Teacher-student relationship is one of "meddler
in the middle," not sage on the stage or guide on
the side
2.84 3 3 1.167
107 Presents at local, state, and national conferences 2.11 2 2 0.994
111 Willingness to share experiences in use of
technology and digital resources with colleagues
2.63 3 3 1.212
Table 57
Comments for Items from Pedagogy Theme Reaching Consensus for Elimination from
Document During Round 2
Item
number Description Comments
90
Cross-curricular education components
Not actionable
Not sure I understand the
question
94 Open-ended question to start the discussion None
96
Teacher-student relationship is one of "meddler
in the middle," not sage on the stage or guide on
the side
Interesting
facilitator is a more appropriate
term
107 Presents at local, state, and national conferences I don’t think presenting is
necessary to demonstrate digital
literacy competence
111 Willingness to share experiences in use of
technology and digital resources with colleagues
Three of the 12 items in the pedagogy theme failed to reach consensus. Those
items were included in Round 3 for reevaluation by the expert panelists. Similar to other
items failing to reach consensus, the items under the pedagogy theme were not observable
or not actionable. The items are described in Table 58.
Table 58
Items from Pedagogy Theme Failing to Reach Consensus for the Document During
Round 2
Item
number Description Mean Median Mode SD
91 Disaggregates data to determine effectiveness of
instruction
3.11 3 3 1.286
92 Innovative pedagogies 3.33 3.5 4 1.328
97 Universal Design for Learning (UDL) Lessons
plans
3.11 3 3 1.100
Delphi Question 3
Four themes emerged from panelist responses to DQ3 during Round 1. The
themes were characteristics, technology integration, digital and/or information literacy,
and instructional activities. The 61 items indicated by expert panelists in Round 1 were
sorted by theme and alphabetized in preparation for panelists to rate using a Likert-type
scale. Participant language was preserved to the greatest extent possible.
Student characteristics theme. The characteristics theme described personality
traits that could be ascribed to students taught in a classroom by a teacher demonstrating
a high level of digital literacy competence. Of the 61 items for DQ3, 12 items were
categorized in the characteristics theme. Of these 12 items, three reached consensus for
inclusion in the walk-through document, as described in Table 59. For these items, one
panelist commented that “engaged’ had too many connotations, and one panelist stated
being open to inquiry-based learning was not observable.
Table 59
Items from Student Characteristics Theme Reaching Consensus for Inclusion in
Document During Round 2
Item
number Description Mean Median Mode SD
156 Empowered learners who ask and answer
questions
3.32 4 4 1.157
157 Engaged learners 3.42 4 4* 1.346
161 Open to inquiry-based learning 3.21 4 4 1.228
Note: Multiple modes exist for item 157 as reported by SPSS. The smallest value is
shown.
Four of the 12 items in the characteristics theme reached consensus for
elimination from the walk-through document. They are described in Table 60. Panelists
noted for each item that they were not observable during a classroom walk-through.
Table 60
Items from Student Characteristics Theme Reaching Consensus for Elimination from
Document During Round 2
Item Description Mean Median Mode SD
number
158 Entrepreneurial spirit where students feel
empowered to follow their ideas and passions
2.53 2 1* 1.264
159 Experimentation with new ideas. 2.84 3 3 1.015
160 Have a lower fear of failure and a greater
risktaking attitude
2.74 3 3 1.147
165 Willing to take the risk of attempting something
at which they may fail
2.94 3 3 1.162
Note: Multiple modes exist for item 158 as reported by SPSS. The smallest value is
shown.
Five of the 12 items in the characteristics theme failed to reach consensus. Those
five items were included in Round 3 for reevaluation by the expert panelists. The items
are described in Table 61. Again, panelists commented these items were either not limited
to digital literacy or were not observable.
Table 61
Items from Student Characteristics Theme Failing to Reach Consensus During Round 2
Item
number Description Mean Median Mode SD
154 Collaborate to solve problems 3.21 3 3 1.134
155 Effective problem solver 3.05 3 4 1.129
162 Persistence 3.37 3 3 1.165
163 Self sufficiency 3.05 3 3 1.224
164 Take risks, model, and try new things 3.32 3 3 1.157
Student technology integration theme. The technology integration theme
encompassed items describing student use of digital tools and how technology was
integrated into instruction in a classroom with a teacher demonstrating a high degree of
digital literacy competence in a classroom. Of the 61 items for DQ3, 24 were categorized
in the technology integration theme. Thirteen of the 24 items reached consensus for
inclusion in the classroom walk-through document. These items are listed in Table 62.
Comments for the items reaching consensus demonstrated some confusion about
several of the items. Several items received comments indicating they were either too
vague or broad. Some items were again called into question as to whether they would be
observable on a walk-through. All comments for items reaching consensus for inclusion
in the walk-through document for the technology integration theme can be found in Table
63.
Of the 24 items in the technology integration theme, two reached consensus for
elimination from the walk-through document. “Begin to have an awareness of the scope
of what is in their hands (mean = 2.33, median = 2.5, mode = 3, SD = 1.029) was
determined to be too vague and confusing according to panelist comments. “Understand
greater implications and evidence in their work” (mean = 2.89, median = 3, mode = 3, SD
= 1.079) caused one panelist to ask, “Greater implications of what?”
Table 62
Items from Student Technology Integration Theme Reaching Consensus for Inclusion in
Document During Round 2
Item
number Description Mean Median Mode SD
113 Ability to find appropriate resources for own
questions
3.47 4 5 1.389
114 Ability to troubleshoot when needed 3.337 4 4 1.212
116 Accessing what teachers have provided for them
using the technologies available
3.32 4 4 1.204
118 Apply their knowledge and ideas with
technology
3.89 4 4 0.875
120 Create products to demonstrate their learning 3.79 4 5 1.273
122 Demonstrate understanding when digital
resources are best and when print resources are
more appropriate
3.79 4 4 1.084
123 Effectively use technology to create, organize,
and access safely curated materials
4.00 4 4 0.816
127 Multiple ways both digital and non-digital for
students to share their knowledge
3.44 4 4 1.199
128 Opportunities to speak, listen, read, write, and 3.53 4 4 1.219
present with technology
131 Storing, using, and creating content in the
classroom and storing online
3.32 4 4 1.250
133 Use a variety of digital tools appropriate for the
product they are going to create
3.58 4 4 1.017
134 Use technology appropriately and effectively 4.05 4 4 1.026
135 Willingness to experiment with new technologies 3.79 4 4 1.032
Nine of the 24 items in the technology integration theme failed to reach
consensus. Those nine items were included in Round 3 for reevaluation by the expert
panelists. The items are described in Table 64. Comments for the technology integration
items failing to reach consensus indicated confusion among expert panelists. Several
panelists asked questions about the items in the comments field. All comments and
questions about these items are listed in Table 65
Table 63
Comments for Items from Student Technology Integration Theme Reaching Consensus for
Inclusion in Document During Round 2
Item
number Description Comments
113 Ability to find appropriate resources for own
questions
Too vague. Digital resources?
114 Ability to troubleshoot when needed They’re not IT professionals ...
they’re digitally-literate students
116 Accessing what teachers have provided for them
using the technologies available
What does this mean?
118 Apply their knowledge and ideas with technology Not sure what this statement
means
120
Create products to demonstrate their learning
Not clear
What kinds of products? Student
products
122 Demonstrate understanding when digital
resources are best and when print resources are
more appropriate
[This] sounds [like it was] written
by a librarian. This is a DIGITAL
scale, about DIGITAL literacy
123 Effectively use technology to create, organize,
and access safely curated materials
None
127 Multiple ways both digital and non-digital for Not actionable Vague
students to share their knowledge
128 Opportunities to speak, listen, read, write, and
present with technology
If available
131
Storing, using, and creating content in the
classroom and storing online
Not actionable
I don’t know what this means
133 Use a variety of digital tools appropriate for the
product they are going to create
They might be proficient with one
that is excellent. If you have
(example) Adobe, why would you
want to use free online tools?
134 Use technology appropriately and effectively It’s a little broad in my mind
135 Willingness to experiment with new technologies May not be observable in a single
session
Table 64
Items from Student Technology Integration Theme Failing to Reach Consensus During
Round 2
Item
number Description Mean Median Mode SD
115 Able to use digital tools with ease 3.68 4 3* 1.003
117 Adept with and can teach others how to use tools 3.32 3 3 1.293
121 Curate resources for themselves and others 3.53 4 3* 1.020
124 Effortlessly self-select and evaluate the best
tool(s) for the job at hand
3.47 4 3* 1.307
125 Empowered by the technology to be
selfactualized learners
3.47 3 3 1.264
126 Enthusiasm for learning how to use new
technologies
3.42 3 3 1.071
129 Proficient in online research 3.53 3 3 1.073
130 Self-directing paths of locating information 3.37 3 3 1.383
136 Work collaboratively with their teacher, peers,
and stakeholders outside their classroom/building
3.00 3 3 1.291
Note: Multiple modes exist for item 115 as reported by SPSS. The smallest value is
shown.
Table 65
Comments for Items from Student Technology Integration Theme Failing to Reach
Consensus During Round 2
Item
number Description Comments
115 Able to use digital tools with ease None
117
Adept with and can teach others how to use
tools
Vague. Technology tools?
What tools?
121 Curate resources for themselves and others What kinds of resources? And
isn’t curation something that a
librarian does, not a student?
124
Effortlessly self-select and evaluate the best
tool(s) for the job at hand
I would hope that most all
students do this.
I’m not sure this is effortless...
It should require effort
125 Empowered by the technology to be
selfactualized learners
I think this is too difficult to
observe in a walk-through
Table 65 (continued)
Item
number Description Comments
126 Enthusiasm for learning how to use new
technologies
Again... enthusiasm... just the
ability
129 Proficient in online research I think this is observable, but in
every class?
130
Self-directing paths of locating information
For whom?
I don’t know what this means.
136 Work collaboratively with their teacher, peers,
and stakeholders outside their
classroom/building
Not observable
Student digital and/or information literature literacy theme. The digital and/or
information literacy theme described how students use digital and information literacy
tools in a classroom taught by a teacher with a high level of digital literacy competence,
including digital citizenship skills. Eight items from panelist responses during Round 1
were grouped under the digital and/or information literacy theme. These were presented
to expert panelists for rating using a Likert-type scale. Of the eight items, five reached
consensus for inclusion in the classroom walk-through document, as described in Table
66.
Despite reaching consensus for inclusion, there were comments about three of the
items. “Demonstrate and practice all facets of digital citizenship at all times” was
objected by one panelist because one “Can’t know if students practice all facets.” It was
mentioned that “Demonstrate appropriate online behavior, understanding the different
types of plagiarism and actively work to create their own materials” was inappropriate
because one panelist did not agree students needed to actively work to create their own
materials. Lastly, “Grasp the concept of copyright and wrongs appropriate for their grade
level” was determined by one panelist to be not observable.
Table 66
Items from Student Digital and/or Information Literacy Theme Reaching Consensus for
Inclusion in Document During Round 2
Item
number Description Mean Median Mode SD
166 Comply with district policies such as the AUP 4.05 4 5 1.224
168 Demonstrate and practice all facets of digital
citizenship at all times
3.89 4 5 1.150
169 Demonstrate appropriate online behavior,
understanding the different types of plagiarism
and actively work to create their own materials
3.84 4 5 1.302
170 Demonstrate respect for intellectual property and
appropriately credit others for their work
4.32 5 5 0.885
171 Grasp the concept of copyright and wrongs
appropriate for their grade level
3.95 4 5 1.079
Of the eight items in the digital and/or information literacy theme, none of the
items reached consensus for elimination from the walk-through document. Three of the
eight items in the digital and/or information literacy theme failed to reach consensus.
Those items were included in Round 3 for reevaluation by the expert panelists. The items
are described in Table 67. One panelist noted the item “Work with peers, teachers, and
parents to obtain digital skills” was “hard to observe.”
Table 67
Items from Digital and/or Information Literacy Theme Failing to Reach Consensus
During Round 2
Item
number Description Mean Median Mode SD
167 Created work includes appropriate citation of
sources 3.79 4 3* 1.032
172 Show a consideration for privacy 3.74 4 3 1.098
173 Work with peers, teachers, and parents to obtain
digital skills
3.47 3 3 1.073
Note: Multiple modes exist for item 167 as reported by SPSS. The smallest value is
shown.
Instructional activities theme. The instructional activities theme contained items
describing the learning-related actions of students in a classroom taught by a teacher with
a high level of competence in digital literacy. Seventeen of the 61 items identified by
expert panelists in Round 1 were grouped under the instructional activities theme. Of
these 17 items, six of them reached consensus for inclusion in walk-through document.
These items are listed in Table 68.
Table 68
Items from Instructional Activities Theme Reaching Consensus for Inclusion in Document
During Round 2
Item
number Description Mean Median Mode SD
137 Able to communicate what they are doing 3.58 4 4 1.017
139 Engage in activities that require high levels of
thinking and encourage collaboration
3.53 3 3 0.964
148 Student-centered learning with outcomes that use
critical thinking and attempting to solve real-
world problems
3.56 4 4 1.381
149 Technology isn’t just an add-on or something
learned in a silo and is instead woven into
lessons to build and grow student understanding
3.89 4 5 1.41
150 Understand different registers/purposes for
digital communication
3.53 3 3 0.905
152 Working on clear academic objectives and
behavioral norms set by the teacher and the
technology being used isn’t the goal in itself
3.47 4 4 1.389
Comments from panelists indicated that while the items reached consensus, there
was some concern that they were not limited to digital literacy. “Engage in activities that
require high levels of thinking and encourage collaboration” even received one comment
beginning with the word, “Ick.” All comments for the items under the instructional
activities theme included in the walk-through document are listed in Table 69.
Table 69
Comments from Items from Instructional Activities Theme Reaching Consensus for
Inclusion in Document During Round 2
Item
number Description Comments
137 Able to communicate what they are doing Not limited to digital technology
139
Engage in activities that require high levels of
thinking and encourage collaboration
Ick. ISTE standards encourage
collaboration. I think this
statement has promise, but needs
refinement Who?
148
Student-centered learning with outcomes that
use critical thinking and attempting to solve
real-world problems
Not limited to digital technology
Perhaps, but needs some
specificity vis-a-vis technology
This is a teacher behavior
149 Technology isn’t just an add-on or something
learned in a silo and is instead woven into
lessons to build and grow student understanding
Not a student behavior . . . this is
a teacher one.
150
Understand different registers/purposes for
digital communication
Who?
Registers is awkward
152
Working on clear academic objectives and
behavioral norms set by the teacher and the
technology being used isn't the goal in itself
I’d like to see an
expectation/presumption that the
teacher is expecting the use of
technology in achieving those
objectives.
Not limited to digital technology
Of the 17 items in the instructional activities theme, nine reached consensus for
elimination from the walk-through document. They are described in Table 70. The
primary reason these items were eliminated, based on panelist comments, was that they
were too vague. There were a number of questions about these items, as listed in Table
71.
Two of the 17 items in the instructional activities theme failed to reach consensus.
“High level of creative expression in a judgment-free zone of learning” (mean = 3.11,
median = 3, mode = 3, SD = 1.278) received two comments from panelists: “for whom?”
and “seems like something the teacher would create, not the student.” “Lessons would
involve a variety of modalities” (mean = 3.32, median = 3, mode = 3, SD = 1.336)
received one comment that may have explained why the item did not reach consensus,
“Digital modalities, or just any modality?” Those two items were included in Round 3 for
reevaluation by the expert panelists.
Table 70
Items from Instructional Activities Theme Reaching Consensus for Elimination from
Document During Round 2
Item
number Description Mean Median Mode SD
138 Ask each other for help rather than relying on the
teacher
2.74 3 3 1.284
140 Evaluation 2.53 3 1* 1.231
141 Have choice in how they do their work as well as
who they work with
2.68 3 3 1.108
144 Questioning 2.72 3 3 1.320
145 Reflection and repetition 2.83 3 3 1.383
146 Scaffolded autonomy 2.95 3 3 1.433
147 Shared discussion 2.84 3 3 1.068
151 Understanding of design cycle/process 2.79 3 3 1.228
153 Write for a global audience 2.47 3 3 0.964
Note: Multiple modes exist for item 140 as reported by SPSS. The smallest value is
shown.
Round 2 Summary
Panelists participating in Round 1 were sent an email on January 15, 2018, with a
link on Surveymonkey.com containing the 173 items identified in Round 1, which can be
found in Appendix B. They were asked to rank these items using a Likert-type scale. The
round closed on January 26, 2018 with 19 expert panelists responding out of the original
26 panelists from Round 1. This was a 73% response rate. Central tendency statistical
analysis in the form of mean, median, mode, and SD was calculated for each item using
the SPSS software to determine consensus as shown in Appendix C.
Table 71
Comments from Items from Instructional Activities Theme Reaching Consensus for
Elimination from Document During Round 2
Item
number Description Comments
138 Ask each other for help rather than relying on
the teacher
Who?
140
Evaluation
What? Why? Vague
Reflection?
Too broad
Vague
What about evaluation?
141
Have choice in how they do their work as well
as who they work with
Unsure how this relates to digital
literacy
Who?
144
Questioning
Vague
Too broad
Vague
145
Reflection and repetition
True of all students, not just
digitally-literate ones
Encourages this? Models this?
Too vague
146
Scaffolded autonomy
Vague Huh?
147
Shared discussion
Vague
Vague
151 Understanding of design cycle/process Not measurable
153
Write for a global audience
Possibly. But where’s the digital
literacy?
It might be a personal letter of
recommendation ...
Of the original 55 items for DQ1, 31 items reached consensus for inclusion in the
definition of a digitally literate teacher, 12 items reached consensus for elimination from
the definition, and 12 items failed to reach consensus. Of the original 57 items for DQ2,
33 items reached consensus for inclusion in the walk-through document, 12 items reached
consensus for elimination from the document, and 11 items failed to reach consensus. For
DQ3, of the original 61 items, 27 of them reached consensus for inclusion in the walk-
through document, 15 items reached consensus for elimination from the document, and
19 items failed to reach consensus. Details of the items for each theme and their
consensus status are included in Table 72.
Table 72
Round 2 Item Consensus Status
Question Theme Total
Items
Consensus
for Inclusion
Consensus for
Elimination
Fail to
Reach
Consensus
Question 1 55
Characteristics 6 2 5
Technology
Integration
15 3 4
Digital and/or
Information
Literacy
6 1 1
Pedagogy 2 2 2
Miscellaneous 2 4 0
Question 2 57
Teacher
Characteristics
1 3 3
Teacher Technology
integration
19 4 4
Teacher Digital and/or
Information Literacy
9 1 1
Pedagogy 4 5 3
Question 3 61
Student
Characteristics
3 4 5
Student Technology
Integration
13 2 9
Student Digital and/or
Information Literacy
5 0 3
Instructional Activities 6 9 2
Delphi Round 3
At the conclusion of Round 2, 42 items failed to reach consensus for inclusion in
the definition or the walk-through document. A survey instrument, shown in Appendix D,
was created of these items asking expert panelists to rate them again using a Likert-type
scale.
1 = Should not be included in the definition or walk-through document
2 = Unimportant for use in the definition or walk-through document
3 = Depends on the circumstances if the current item should be included in the
definition or walk-through document
4 = Important to include in the definition or walk-through document
5 = Essential to include in the definition or walk-through document
Timeline for Round 3
An email was sent to the 19 expert panelists completing Round 2 of the current
study on February 19, 2018. Included in the email was a link to a secure Google form
containing the 131 items reaching consensus for either inclusion or elimination for each
of the three Delphi questions found in Appendix D. Expert panelists were asked to
examine the items that reached consensus for inclusion and elimination, then proceed to
re-evaluate the 42 items not reaching consensus. A link to the third round containing the
questionnaire with these 42 items not yet achieving consensus was also included in the
email.
A reminder email was sent on February 28, 2018 to panelists who had not
completed the survey as of that date. Round 3 of the current study concluded on March 5,
2018.
Round 3 Analysis and Results
A total of 18 of the 19 panelists completed Round 3. At the conclusion of Round
3, central tendency statistics of mean, median, mode, and SD was calculated using SPSS
software for each of the items. After calculating the statistics, it was determined that 10 of
the 42 items reached consensus to be included in the definition or walk-through
document. For the purposes of the current study, consensus on an item required the item
to meet one or more of the following criteria in order to be included in the definition or in
the walk-through document:
•70% or greater panel agreement
•4.0 or greater mean score
•4.0 or greater median and mode score
•1.0 or less standard deviation score
At the conclusion of Round 3, 13 of 42 items reached consensus for elimination
from the definition and the document. In order to reach consensus, an item met one or
more of the following criteria in order to be eliminated in the definition or the
walkthrough document:
•70% or greater panel agreement
•2.99 or less mean score
•2.99 or less median and mode score
•1.0 or less standard deviation score
At the conclusion of Round 3, 19 of the 42 items failed to reach consensus. A
reliability analysis was conducted on the 42 items. A Cronbach’s alpha statistical analysis
was performed to determine reliability and internal consistency of the Likert-type scale
provided to expert panelists during Round 3. Cronbach’s alpha showed the questionnaire
to reach acceptable reliability, α = 0.978. “The closer Cronbach’s alpha coefficient is to
1.0 the greater the internal consistency of the items in the scale.” (Gliem & Gliem, 2003,
p. 87).
Delphi Question 1
Five themes emerged from the panelist responses to DQ1 during the first round of
the current study. These themes were characteristics, technology integration, digital
and/or information literacy, pedagogy, and miscellaneous. The 12 items that failed to
reach consensus in Round 2 were sorted by theme and alphabetized in preparation for
panelists to rate using a Likert-type scale during Round 3.
Characteristics theme. The characteristics theme described personality traits that
could be ascribed to the definition of a digitally literate teacher. Of the 12 items from
DQ1, five items under the characteristics theme failing to reach consensus in Round 2
were presented to expert panelists to rank with a Likert-type scale.
There was one item under the characteristics theme reaching consensus for
inclusion in the definition at the conclusion of Round 3. “Ethics” (mean = 3.56, median =
4, mode = 5, SD = 1.464) was deemed important enough to include in the definition of a
digitally literate teacher. Strangely, the only comment on this item indicated one panelist
felt the opposite: “Ethics is important but not germane to this.” One other panelist
questioned if ethics was observable. Another panelist commented, “I think it’s important.
But I also think it’s a term with a varied definition.”
Three of the five items reached consensus to be eliminated from the definition
after Round 3. Information about these three items is included in Table 73. Each of the
eliminated items received at least two negative comments explaining panelist rationale
for deciding to eliminate the item. All comments from expert panelists regarding the
items eliminated are described in Table 74.
Table 73
Items from Characteristic Theme Reaching Consensus for Elimination from Definition
During Round 3
Item
number Description Mean Median Mode SD
1 Always-learning mentality 2.33 2 1 1.237
3 Growth mindset 2.89 3 4 1.410
4 Not afraid to try even if they’re pretty sure
they’ll fail on the first attempt
2.67 2.5 1 1.414
Table 74
Comments for Items from Characteristic Theme Reaching Consensus for Elimination
from Definition During Round 3
Item
number Description Comments
1
Always-learning mentality
Similar to the growth mindset
How can I observe someone’s
mentality towards learning?
not specific to digital literacy; not
demonstrable
3
Growth mindset
A teacher should have this in
general. I don’t think it defines a
techie teacher.
Encompasses always learning but
with more specificity.
4 Not afraid to try even if they’re pretty sure
they’ll fail on the first attempt
Again, I’m not sure this is techie.
Similar to growth mindset
One item, “Strong desire to put students first,” failed to reach consensus for
inclusion or elimination from the definition (mean = 3.17, median = 3, mode = 3, SD =
1.339). Comments for this item included reasons from panelists such as “Every teacher
should likely have this” and “Desire is not observable.”
Technology integration theme. The technology integration theme described how
a digitally literate teacher evaluated and used technology and integrated digital tools into
instruction. Of the 12 items from DQ1 presented for panelists to re-evaluate during
Round 3, four items were in the technology integration theme. Of those, one item
achieved consensus for inclusion in the definition of a digitally literate teacher. “Proper
use of technology, especially in front of students” (mean = 3.44, median = 4, mode = 4,
SD = 1.315) was deemed important enough to include in the definition. There were no
panelist comments related to this item.
Three of the four items under the technology integration theme failed to reach
consensus. They are included in Table 75. Expert panelists reacted negatively to these
items, as indicated in Table 76. Panelists determined these items either needed to be
altered or were already encompassed in other items that had reached consensus.
Table 75
Items from Technology Integration Theme Failing to Reach Consensus During Round 3
Item
number Description Mean Median Mode SD
6 Insatiable curiosity for new applications/
curriculum tools which is then incorporated
into curriculum/training
3.18 3 3 1.286
7 Knowledge of many digital tools and resources 3.33 3 3 1.283
12 Understanding of SAMR model 3.22 3 2* 1.353
Note: Multiple modes exist for item 12 as reported by SPSS. The smallest value is shown.
Digital and/or information literacy theme. The digital and/or information literacy
theme described how a digitally literate teacher uses digital and information skills in
instruction. This includes digital citizenship instruction. Expert panelists were presented
with one item in this theme failing to reach consensus in Round 2. The item, “Skilled at
being able to identify and synthesize information” (mean = 3.50, median = 4, mode = 4,
SD = 1.249) reached consensus in Round 3 despite one panelist describing the item as
“vague.”
Table 76
Comments for Items from Technology Integration Theme Failing to Reach Consensus
During Round 3
Item
number Description Comments
6
Insatiable curiosity for new applications/
curriculum tools which is then incorporated into
curriculum/training
I think you can want new tools but
not technology tools.
I think this can be demonstrated by
items that have been included.
Learning for learning sake does
not translate to being a better
teacher
I would remove... insatiable
curiosity... replace with research -
and the teachers need training, too
Difficult to measure curiosity
7
Knowledge of many digital tools and resources
I don’t think this is necessary. I
don’t know all of them.
I think this can be demonstrated
by items that have been included.
Again, does not necessarily
translate into being a better teacher
12
Understanding of SAMR model
SAMR is a person’s dissertation. I
don’t know that it’s a
requirement.
This is important, however on the
consensus results SAMR was
placed under consensus for
exclusion
Pedagogy theme. The pedagogy theme contained items describing the ability of a
digitally literate teacher to appropriately apply pedagogy in a digital environment. There
were two items grouped under the pedagogy theme presented to the expert panelists that
failed to reach consensus in Round 2. Of those items, one reached consensus to be
included in the definition of a digitally literate teacher. “Knowledgeable about current
strategies” (mean = 3.35, median = 4, mode = 4, SD = 1.320) was included in the
definition despite panelist comments the item was “vague” or “unclear.”
The second item presented to expert panelists failed again to reach consensus.
“Student centered, standards-aligned instruction (mean = 3.25, median = 3, mode = 3, SD
= 1.227) was not deemed important enough for inclusion in the definition of a digitally
literate teacher. While one panelist commented the item was “observable,” another
commented, “All teachers should want this.”
Delphi Question 2
Four themes emerged from the panelist responses to DQ2. The themes were
characteristics, technology integration, digital and/or information literacy, and pedagogy.
The 11 items for DQ2 failing to reach consensus in Round 2 were sorted by theme and
alphabetized for expert panelists to rate again using a Likert-type scale during Round 3.
Teacher characteristics theme. The characteristics theme described personality
traits that could be ascribed to a teacher demonstrating a high level of digital literacy
competence in a classroom. Three of the 11 items for DQ2 were grouped under the
characteristics theme. Two of these items reached consensus for elimination from the
walk-through document. “Continuously seeking new knowledge” (mean = 2.39, median =
2, mode =1, SD = 1.243) received a comment explaining why one panelist decided to
eliminate the item: “Any good teacher should be doing this.” “Lifelong learner” (mean =
2.67, median = 2, mode = 1, SD = 1.455) was eliminated by panelists according to
comments because “Any good teacher should be doing this” and the item was “Not
observable.”
One item, “Willingness to take risks,” failed to reach consensus (mean = 3.28,
median = 3.5, mode = 4, SD = 1.227). Comments on this item included “I think this is
likely true, but hard to observe in a walk-through” and “observable—primarily through
failure thus ... learning.”
Teacher technology integration theme. The technology integration theme
described use of digital tools and how technology was integrated into instruction by a
teacher demonstrating a high degree of digital literacy competence in a classroom. Of the
four items in the technology integration theme failing to reach consensus in Round 2,
none of them reached consensus for inclusion in the walk-through document.
Two of the items from the technology integration theme reached consensus for
elimination from the document. “Ensures that students use the technology for learning
more than he/she does for instruction” (mean = 2.72, median = 3, mode = 3, SD = 1.227)
and “Leadership role in encouraging use of technology by all members of the learning
community” (mean = 2.83, median = 3, mode = 3, SD = 1.200) were included in the
document. One panelist commented on the first item, “I’m OK with either, frankly.”
Comments for the second item included “No. You don’t need to be a leader to evidence
tech use” and “Leaders often say one thing, then do another when central office leaves.”
Two of the items failed again to reach consensus. “Expands the walls of the
classroom” (mean = 3.06, median = 3, mode = 3, SD = 1.211) was described as “vague”
by one of the expert panelists. “Uses technology regularly and in a way that goes beyond
substitution level of SAMR model” (mean = 3.06, median = 3, mode = 2, SD = 1.349)
was eliminated, according to panelists comments because it was “Evidenced by other
items included” and “SAMR is a dissertation and in some cases is subjective.”
Teacher digital and/or information literacy theme. The digital and/or
information literacy theme described how a digitally literate teacher used digital and
information skills in instruction, including digital citizenship instruction. Expert panelists
were presented with one item under the digital and/or information literacy theme failing
to reach consensus in Round 2. The item, “Works with parents explaining the legal and
ethical laws and expectations associated with working in a digital learning environment”
(mean = 3.17, median = 3, mode = 3, SD = 1.425) failed to reach consensus again in
Round 3. A panelist noted this item was a “Great idea! But hard to observe in a
walkthrough.”
Pedagogy theme. The pedagogy theme contained items describing the ability of a
teacher with a high level of competence in digital literacy to appropriately apply the
application of pedagogy in a digital environment. The three items grouped under the
pedagogy theme failing to reach consensus in Round 2 were presented to expert panelists
for rating again using a Likert-type scale in Round 3. Of these three items, two of them
reached consensus for elimination from the document. “Disaggregates data to determine
effectiveness of instruction” (mean = 2.94, median = 3, mode = 3, SD = 1.162) were
eliminated in part because panelists commented “Any good teacher should be doing this”
and “As you move forward, a list of things gathered in the data is important.” “UDL
lesson plans” (mean = 2.72, median = 2, mode = 2, SD = 1.364) was rated for elimination
because of the following explanation from a panelist: “Any good teacher should be doing
this.” In contrast, one panelist was in favor of including the item by commenting,
“tangible and observable—I have always used this mindset, but I have never used a form.
Excellent idea!”
One item, “Innovative pedagogies” (mean = 3.06, median = 3, mode = 4, SD =
1.305) failed again to reach consensus. Rationales presented by panelists included
“Innovative doesn’t mean technological” and “These change often, a list should be on the
form to circle—which would require the observer to be able to identify the one being
employed.”
Delphi Question 3
Four themes emerged in DQ3. The themes were characteristics, technology
integration, digital and/or information literacy, and instructional activities. The 19 items
failing to reach consensus in Round 2 were sorted by theme and alphabetized for panelists
to rate using a Likert-type scale.
Student characteristics theme. The characteristics theme described personality
traits that could be ascribed to students taught in a classroom by a teacher demonstrating
a high level of digital literacy competence. Of the 19 items for DQ3, five items were
grouped under the characteristics theme and presented to the expert panelists for
reevaluation. Of these five items, “Effective problem solver” (mean = 3.44, median = 4,
mode = 4, SD = 1.294) reached consensus for inclusion in the walk-through document
despite one panelist referring to the item as “vague.”
One of the five items, “Self-sufficiency” (mean = 2.83, median = 3, mode = 2, SD
= 1.295), reached consensus for elimination from the walk-through document. One
panelist explained, “Self-sufficiency can be bad . . . you might be continuing to do the
wrong things.”
Three items in the characteristics theme failed again to reach consensus in Round
3. “Collaborate to solve problems” (mean = 3.22, median = 3, mode = 2, SD = 1.350) was
called “vague” by one panelist. “Persistence” (mean = 3.11, median = 3, mode = 3, SD =
1.230) was also vague and “not limited to digital literacy” according to panelist
comments. “Take risks, model, and try new things” (mean = 3.28, median = 3.5, mode =
4, SD = 1.179) was “hard to observe” and “not limited to digital environment” according
to panelist comments.
Student technology integration theme. The technology integration theme
encompassed items describing student use of digital tools and how technology was
integrated into instruction in a classroom with a teacher demonstrating a high degree of
digital literacy competence in a classroom. The nine items from the technology
integration theme failing to reach consensus in Round 2 were presented to the expert
panelists to rank again using a Likert-type scale in Round 3. Two of the items reached
consensus for inclusion in the walk-through document. “Adept with and can teach others
how to use tools” (mean = 3.50 median = 4, mode = 4, SD = 1.200), despite inclusion,
generated the comments “What tools? When?” and “Implied by other items included.”
“Self-directing paths of locating information” (mean = 3.24, median = 4, mode = 4, SD =
1.393) likewise received less than positive comments of “vague” and “Not sure what this
statement means.”
One item reached consensus for elimination from the walk-through document in
Round 3. “Work collaboratively with their teacher, peers, and stakeholders outside their
classroom/building” (mean = 2.83, median = 3, mode = 3, SD = 1.150) was eliminated
according to panelist comments because “Any good teacher should be doing this” and
“can you see this on a walk-through—how old are the kids?”
Of the nine items in the technology integration theme failing to reach consensus in
Round 2, six of the items failed again to reach consensus in Round 3. They are described
in Table 77.
Table 77
Items from Student Technology Integration Theme Failing to Reach Consensus During
Round 3
Item
number Description Mean Median Mode SD
24 Able to use digital tools with ease 3.56 3.5 3 1.097
26 Curate resources for themselves and others 3.11 3 3 1.183
27 Effortlessly self-select and evaluate the best
tool(s) for the job at hand
3.44 3.5 5 1.542
28 Empowered by the technology to be
selfactualized learners
3.17 3.5 4 1.505
29 Enthusiasm for learning how to use new
technologies
3.22 3 3 1.263
30 Proficient in online research 3.18 3 5 1.510
Panelists noted in the comments for these items that the items were not crucial to
the walk-through document due to being vague, confusing, or hard to observe. All
comments for the items can be found in Table 78.
Student digital and/or information literature literacy theme. The digital and/or
information literacy theme described how students used digital and information literacy
tools in a classroom taught by a teacher with a high level of digital literacy competence
including digital citizenship skills. The three items grouped under the digital and/or
information literacy theme failing to reach consensus in Round 2 were presented to expert
panelists for rating again using a Likert-type scale. All three items reached consensus for
inclusion in the walk-through document during Round 3. Despite being included in the
document, comments included on “Show a consideration for privacy” and “Work with
peers, teachers, and parents to obtain digital skills” were found by one panelist to be hard
to observe. The items reaching consensus in Round 3 are described in
Table 79.
Table 78
Comments for Items for Student Technology Integration Them Failing to Reach
Consensus During Round 3
Item
number Description Comments
24 Able to use digital tools with ease I think this needs to be more
precise. “With ease” is rather
vague.
26 Curate resources for themselves and others I think an instructional
technologist does this.
27 Effortlessly self-select and evaluate the best
tool(s) for the job at hand
Hard to observe in a
walkthrough.
28
Empowered by the technology to be
selfactualized learners
What does that mean? Can this
actually be observed in a
walkthrough?
Not sure what this statement
means
29 Enthusiasm for learning how to use new
technologies
Nice, but not essential
30 Proficient in online research May not observe in a
walkthrough.
Table 79
Items from Student Digital and/or Information Literacy Theme Reaching Consensus
During Round 3
Item
number Description Mean Median Mode SD
40 Created work includes appropriate citation of
sources
3.39 4 5 1.614
41 Show a consideration for privacy 3.35 4 5 1.455
42 Work with peers, teachers, and parents to obtain
digital skills
3.39 4 4 1.378
Instructional activities theme. The instructional activities theme contained items
describing the learning-related actions of students in a classroom taught by a teacher with
a high level of competence in digital literacy. Two items grouped under the instructional
activities theme failed to reach consensus in Round 2. These items were presented again
to expert panelists for rating using a Likert-type scale. Both of the items reached
consensus for elimination from the document in Round 3. “High level of creative
expression in a judgment-free zone of learning” (mean = 2.67, median = 3, mode = 3, SD
= 1.085) was eliminated according to one panelist because it “Doesn’t have much to do
with technology.” “Lessons would involve a variety of modalities” (mean = 2.72, median
= 2.5, mode = 1, SD = 1.447) was vague. One panelist also cited a link as to why the item
was not appropriate: “To mark Brain Awareness Week this month, 30 internationally
respected neuroscientists, psychologists, and educators issued a public letter asking
teachers to stop wasting time with it. http://thefederalist.com/2017/03/22/brain-
scientistslearning-styles-like-auditory-visual-and-kinesthetic-are-bunk/”
Round 3 Summary
At the conclusion of Round 2, 42 items failed to reach consensus for inclusion in
the definition or in the walk-through document. A survey instrument, shown in Appendix
D, was created of these items asking expert panelists to rate them again using a Likerttype
scale. During Round 3, 10 of the 42 items reached consensus to be included in the
definition or walk-through document. At the conclusion of Round 3, 13 items reached
consensus for elimination from the definition and the document and 19 items failed to
reach consensus. Table 80 provides a summary of items at the conclusion of Round 3.
Table 80
Round 3 Item Consensus Status
Question Theme Total Items Consensus
for Inclusion
Consensus for
Elimination
Fail to
Reach
Consensus
Question 1 12
Characteristics 1 3 1
Technology
Integration
1 3 0
Digital and/or
Information
Literacy
1 0 0
Pedagogy 1 0 1
Miscellaneous 0 0 0
Question 2 11
Teacher
Characteristics
0 2 1
Teacher
Technology
integration
0 2 2
Teacher Digital
and/or
Information
Literacy
0 0 1
Pedagogy 0 2 1
Question 3 19
Student
Characteristics
1 1 3
Student
Technology
Integration
2 1 6
Student Digital
and/or
Information
Literacy
3 0 0
Instructional
Activities
0 2 0
Items failing to reach consensus after three rounds of the current study were
eliminated from consideration for the definition and the walk-through document to
prevent panelist fatigue. It has been recommended that two or three rounds of a Delphi
study are sufficient for this reason (Hsu & Sandford, 2007; Verbeke, 2014). The current
study was completed in three iterations or four rounds.
Delphi Round 4
When consensus was reached on the essential elements for a definition and the
look-fors for the walk-through form, the mock-up of the definition and the form was sent
to expert panelists during Round 4. These can be found in Appendices F and G. Panelists
were given the option to approve the form and definition as written, approve the form and
definition with modifications, or to disapprove the form and definition. The form and
definition were presented to panelists separately, so approval of one was not contingent
on approval of the other.
Timeline for Round 4
An email was sent to the 26 expert panelists initially completing Round 1 on
March 27, 2018. The email expressed gratitude to panelists for their contributions and a
link to a secure Google document, shown in Appendix E, containing a summary of items
that reached consensus for inclusion or exclusion as of the end of Round 3. Panelists were
asked to examine this document prior to completing Round 4. A link to the Round 4
survey instrument on the Surverymonkey.com website containing three questions was
also included in the email. Data collection for Round 4 concluded on April 6, 2018.
Round 4 Analysis and Results
Of the 26 panelists receiving the email invitation, 20 completed Round 4, a 77%
response rate.
Research Question 1
RQ1 was “What qualities does a teacher possess and demonstrate that indicate a
high proficiency in digital and web literacy?” Responses to RQ1 were used to create a
definition of a digitally literate teacher through analyzing panelist responses to Delphi
Question 1. These are reflected in Appendix F.
Delphi Question 1. DQ1 was “What qualities does a teacher possess and
demonstrate that indicate a high proficiency in digital and web literacy?” Panelists were
asked to view the definition of a digitally literate teacher, found in Appendix F, created as
a result of their responses during the first three rounds.
The definition is as follows:
A digitally literate teacher is one who demonstrates proficiency in technology
integration, digital/information literacy skills, and pedagogy in an online environment.
The digitally literate teacher possesses the following characteristics:
●the ability & willingness to search for answers to questions;
●ethics;
●persistence when implementing technology; grit;
●problem-solving skills;
●risk-taking; and
●a willingness to fail.
In regard to technology integration, the digitally literate teacher:
●is fluent with technology—displays a positive attitude and affinity towards
technology, is able to explain technology to others, explores and troubleshoots
technology-related issues;
●frequently uses technology and digital resources in meaningful ways;
appropriately embeds technology into curriculum as a tool to enhance
learning;
●provides students with personalized learning, choice, and ownership through
technology;
●promotes digital equity to prevent educational gaps among students; and
●models leadership and appropriate use of technology for students and
colleagues.
The digitally literate teacher demonstrates an understanding and commitment to the
principles of digital citizenship, digital literacy, and information literacy by:
●experimenting with and learning about new web/digital literacies;
●focusing on high level, critical thinking while instructing students in digital
citizenship, digital and web literacy, research skills, and copyright law; and
●locating, evaluating, synthesizing, and using information effectively and
efficiently.
A digitally literate teacher demonstrates an understanding of technology-related
pedagogy through:
●assuming varied roles in the learning process with digital age learners such as
that of facilitator, learner, supporter, and challenger;
●practicing current instructional technology strategies and best practices;
●combining knowledge of subject matter and integration of technology tools in
a manner beneficial to all learners; and
●integrating ISTE and state technology standards into instruction.
Panelists were asked to review the definition and select one of the following
options:
●accept the definition of a digitally literate teacher as written; ● do not accept
the definition of a digitally literate teacher as written; or ● accept the
definition of a digitally literate teacher with modification.
Panelists selecting the third option were requested to provide the suggested
modification in the comments field following the question.
Of the three options, 66.7% of the panelists (14 of 21) selected the first option, to
accept the definition of a digitally literate teacher as presented. Another 28.57%, or six
panelists, selected to accept the definition with modifications and one panelist chose to
reject the definition. The percentages are demonstrated in Figure 6.
Panelist Responses to Proposed Definition
80%
70%
60%
50%
40%
30%
20%
10%
0%
I accept the definition of a I do not accept the definition of a I accept the definition of a
digitally literate teacher as digitally literate teacher as digitally literate teacher with
written written modification (please add
suggested modification in the
comment field below)
Figure 6. Panelist responses to proposed definition.
Panelists were asked to provide a comment to recommend modifications if they
selected Option 3, “I accept the definition of a digitally literate teacher with
modification.” There were nine comments made by panelists. Two of them were
congratulatory and seven comments contained modifications.
● Great work!
● In regard to technology integration, the digitally literate teacher (additions
below):
Possesses depth of knowledge of content, pedagogy, and technology;
Understands that students must develop their technology skills, and provides
appropriate opportunities for them to use technology for learning.
● I think the second set of bullets is really the “money” here. I appreciate most
of what’s been written, but that second set seems to be the good stuff.
● A digitally literate teacher is one who demonstrates proficiency in technology
integration, digital/information literacy skills, and pedagogy. Remove the
words “in an online environment” as you don’t need to be online for these
skills.
● I appreciate the way this process played out; I think this definition will be
widely applicable.
● “Risk Taking” and “Willingness to Fail” seem redundant as standalone
qualities. Combining them into one sentence of just using “Risk Taking”
seems appropriate.
● RQ1 states “digital and web literacy,” but your definition seems to indicate
that it is only for “online” in “technology integration, digital/information
literacy skills, and pedagogy in an online environment.” I would suggest the
following: “in technology integration, digital/information literacy skills, and
online pedagogy.”
● I suggest that a digitally literate teacher is familiar with models such as
SAMR and TPACK to understand the cognitive demand of tasks and
appropriate usage of digital tools.
● Pedagogy should be first in the definition before anything else.
Research Question 2
RQ2 was “Which of these behaviors could be observed on a walk-through
observation?” RQ2 was addressed through panelist answers to DQ2 and DQ3. This data
was formulated into a proposed walk-through document, which was presented to expert
panelists in Round 4.
Delphi Question 2 (DQ2) “What are the observable behaviors of a teacher with a high
level of digital literacy competence?”
Delphi Question 3 (DQ3): “What are the observable behaviors of students in a
classroom with a teacher demonstrating a high level of digital literacy
competence?”
Research Question 3
The document presented to expert panelists in Round 4 attempted to address RQ3,
which was: “What are the elements of such a form?” A link to the Proposed Stout Digital
Literacy Walk-through document was presented to panelists. This document can be found
in Appendix G. The document was stored in a secure online Google Drive environment
and was included as a link in the fourth round of the panelist survey. The contents of the
document are in Figures 7 and 8.
Proposed Stout Digital Literacy Walk-through Document
A digitally literate teacher is one who demonstrates proficiency in technology integration,
digital/ information literacy skills, and pedagogy in an online environment.
Students in a digitally literate teacher’s class are empowered and engaged learners who
effectively use technology to answer questions, solve problems, and demonstrate
knowledge.
Teacher and student behaviors are organized into three domains: technology integration,
digital/information literacy, and pedagogy/instructional activities. This organization
allows for a focused observation of one domain or a holistic observation of all three
domains.
Technology Integration
The teacher: The students: Notes/Observations
• Effectively integrates
technology for critical
thinking, creativity, and
productivity
• Introduces technologyrelated
current events when
appropriate to demonstrate
the impact of technology on
everyday life
• Encourages student choice
and exploration in
technology applications
• Differentiates content,
process, and products using
technology
• Shares experiences and
digital resources with
colleagues
• Uses varied resources in
whole class, small group,
and individual student
settings in engaging and
meaningful ways
•
•
•
•
•
•
•
•
•
Locate resources to find and
use information efficiently
Teach others how to use
tools and troubleshoot
Choose appropriate
resources to create products
demonstrating learning
Effectively use technology to
create, organize, and access
safely curated
materials
Share knowledge in multiple
ways
Speak, listen, read, write,
and present with technology
Store, use, and create
content online
Use a variety of digital tools
appropriate for products
they create
Experiment with new
technologies
Figure 7. Page 1of proposed Stout Digital Literacy Walk-through Document presented to
expert panel during Round 4.
Digital/Information Literacy
The teacher: The students: Notes/Observations
•
•
•
•
•
•
•
•
Demonstrates copyright, digital
citizenship, and research skills
Models online communication and
computational thinking skills
Appropriately uses social media
and global communication tools in
class
Finds teachable moments to
model legal and ethical behaviors
and expectations associated with
online access
Verifies Terms of Service and use
of student data for digital
resources
Familiarizes self with federal
mandates, district AUP, and other
policies regarding safe and
effective use of digital resources
Integrates digital literacies in
lessons
Credits image/video/music
sources and requires the
same from students
•
•
•
•
•
Comply with federal copyright
laws and district policies such as
the AUP
Practice digital citizenship at all
times
Demonstrate appropriate online
behavior, respect for intellectual
property, and appropriately credit
others for their work
Show a consideration for digital
privacy
Work with peers, teachers, and
parents to obtain digital skills
Pedagogy/ Instructional Activities
The teacher: The students: Notes/Observations
•
•
•
•
•
•
Uses a variety of teaching
strategies to engage students at all
developmental levels
Designs high quality
assessments to check for
understanding Acts as facilitator,
providing students access to
tools and resources in a manner
that lets students be the experts
Allows students to learn from
mistakes and choose how to
synthesize learning
Uses technology to provide tiered,
personalized instruction and
scaffolding
Continually learns new strategies
and tools; models their
•
•
•
•
•
•
Communicate what they are doing
Are engaged in activities that
require high levels of thinking and
encourage collaboration
Apply inquiry-based learning
Understand different
registers/purposes for digital
communication
Use technology and critical
thinking to solve real-world
problems and demonstrate
learning
Work within clear academic
objectives and behavioral norms
set by the teacher
implementation
Figure 8. Page 2 of proposed Stout Digital Literacy Walk-through Document.
The expert panelists reviewed the document and chose one of three options.
● Accept the walk-through document as written
● Do not accept the walk-through document as written
● Accept the walk-through document with modification
Of the 26 recipients of the document, 20 panelists responded. Half of the panelists
accepted the document as written, seven (35%) accepted the document with modification.
Three panelists (15%) did not accept the walk-through document. Figure 9 illustrates the
responses.
Figure 9. Panelist responses to proposed walk-through document
Panelists selecting the third option were requested to provide suggested
modifications in the comments field following the question. Comments listed by panelists
recommended the following modifications:
I accept the walkthrough
document as written
I do not accept the walkthrough
document as written
I accept the walkthrough
document with modification
(please add suggested
modifications in the comment
field below)
%0
10%
20%
%30
40%
50%
60%
Participant Responses to Proposed Walk-through Document
● Consider adding more elements from the ISTE Student and Educator
Standards.
● I do not believe you can observe some of the items listed, you would need to
interview the teacher. The term walk-through document might not be the
correct term for this document.
● I don’t think some of these are observable in a single walk-through. For
example, I might not be able to observe a teacher finding a “teachable moment
to model legal and ethical behaviors” in a single walk-through. I’m wondering
if there ought to be some scale for these. For example, at least three of the
following five behaviors.
● This document assumes previous knowledge which may not be applicable to
the administrator(s) participating in walk-through. Links to definitions,
training modules, f2f [face to face] training will need to be included in this
model prior to distribution of document.
● I think the document itself needs some design work. Grouping similar
demonstrable activities together would make it easier to work through. Also
separate documents for Teacher/Student might be more beneficial so the
observer can focus on one area at a time.
● I don’t think an arbitrary line is purposeful in this situation. Showing all items
in a rank format would be more helpful. For example, SAMR model would be
very helpful observable behavior, but it is off the list. It may have a lower
ranking, but it shouldn’t be eliminated.
● I’d suggest adding a qualifier to students practicing digital citizenship such as
“good” or “appropriate” digital citizenship. This looks great!
● Pedagogy first.
An additional question posed to panelists in Round 4 was “Who do you think
would use this document? Please select all that apply.” Options presented to panelists
included:
● Campus principal or assistant principal
● Central office administrator
● Instructional coach, specialist, facilitator
● Other (please specify)
The largest number of respondents selected instructional coach, specialist,
facilitator (19), followed by campus principal or assistant principal and central office
administrator, both of which received 15 votes. Six panelists selected the “other” option.
Three of the panelists recommended teachers. Technology vendors, teachers, curriculum
coordinators, parents. and students were each mentioned once. A frequency graph can be
found in Figure 10.
Round 4 Summary
The 26 original expert panelists were emailed the link to the Round 4 survey for
the current Delphi study. Round 4 consisted of a mock-up of the definition and of the
digital literacy walk-through document found in Appendices F and G. Participants were
given the option to approve the form and/or the definition as written, approve the form
and/or the definition with modifications, or to disapprove the form and/or the definition.
The form and definition were presented to participants separately, so approval of one was
not contingent on approval of the other.
Figure 10. Frequency of Document Users According to Panelists.
Of the 26 panelists receiving the email invitation, 21 panelists rated the definition
of a digitally literate teacher, and 20 completed the entire Round 4, which was a 77%
response rate. Twenty of the 21 panelists accepted the definition of a digitally literate
teacher either as written or with modifications. Seventeen of 20 panelists accepted the
walk-through document either as written or with modifications. Each of the panelists
indicated at least one possible group that would be able to make use of the walk-through
document, the most highly selected being an instructional coach, specialist, or facilitator.
Summary
The current study utilized a four-round Delphi method to ascertain the traits to
compose a definition of a digitally literate teacher and the demonstrable behaviors of
students and the teacher in a classroom with a teacher possessing a high degree of digital
literacy. The current study sought to create a definition of a digitally literate teacher and a
non-evaluative document identifying possible areas of digital literacy deficiency in
Campus principal or
assistant principal
Central office
administrator
Instructional coach,
specialist, facilitator
Other (please specify)
0
2
4
6
8
10
12
14
16
18
20
Frequency of Document Users According to Panelists
teachers and areas at which teachers excel. The research questions the current study
addressed are as follows:
Research Question 1 (RQ1) What are the indicators of digital literacy competency in
teachers as identified by a panel of experts?
Research Question 2 (RQ2) Which of these behaviors could be observed on a
walkthrough observation?
Research Question 3 (RQ3) What are the elements of such a form?
During Round 1 of the current Delphi study, 26 expert panelists completed three
questions:
Delphi Question 1 (DQ1) What qualities does a teacher possess and demonstrate that
indicate a high proficiency in digital and web literacy?
Delphi Question 2 (DQ2) What are the observable behaviors of a teacher with a high
level of digital literacy competence?
Delphi Question 3 (DQ3) What are the observable behaviors of students in a classroom
with a teacher demonstrating a high level of digital literacy competence? DQ1
corresponded with RQ1 and the information obtained during the four rounds of the
current study were intended to produce a definition of a digitally literate teacher.
These results are shown in Appendix F.
DQ2 and DQ3 were initially posed to panelists in order to answer RQ2 in Rounds
1-3 of the current study. Information gleaned from panelist responses to DQ2 and DQ3
were used to create the Stout Digital Literacy Walk-through Form shown in Appendix G,
which addressed RQ3.
Chapter 5 will address discussion, implications, and recommendations based on
the definition and walk-through document that were created throughout the four-round
Delphi study.
CHAPTER 5. DISCUSSION, IMPLICATIONS, RECOMMENDATIONS
Students are adept at making personal communications via the Internet, yet they
typically are not proficient in the application of digital literacy skills in an academic
setting (Li & Ranieri, 2010; Stout, 2017). Students need to be able to use information
critically—to “evaluate resources carefully and determine how to use relevant
information to solve problems and make wise decisions” (Farmer, 2011, p. 387). The
need to adequately evaluate information is mostly going unmet by public schools
(Alexander et al., 2016; Johnson, 2007; Lyons, 2010).
Improving digital literacy skills was included in the 2017 Horizon Report from the
New Media Consortium and Consortium for School Networking as one of the six most
significant challenges facing technology integration in schools today. Digital literacy was
defined as “the ability to use information and communication technologies to find,
understand, evaluate, create, and communicate digital information, an ability that requires
both cognitive and technical skills” (American Library Association, 2013, p. 2).
According to Poore (2011), there are six components to digital literacy—
accessing information, managing information, evaluating information, creating new
understandings, communicating with others, and using information and communication
technologies appropriately. Application of digital literacy skills encompass
“troubleshooting abilities, purposes for connecting, skills in using popular technology
tools, as well as communication literacy, and web literacy” (Blummer, 2008, p. 38).
Digital literacy “requires a very specific set of educator knowledge and teaching
skills compared to other goals currently set under the digital citizenship umbrella” (Jones
& Mitchell, 2015, p. 2). Hobbs (2010) identified some additional reasons for teachers not
creating instruction in the use of digital literacy skills such as an unfamiliarity of the
technology, a lack of understanding of effective technology integration, or prioritizing
high stakes testing superseding 21st Century skills such as digital literacy. The current
study attempted to identify essential behaviors of digitally literate teachers so the
identified behaviors can be used as a model for professional learning for all classroom
teachers.
Few districts offer professional learning on topics such as
•choosing the right tool to find, use, or create information;
•finding a person online, for example an expert in a certain discipline, and
establishing contact details;
•establishing who owns information and ideas found online;
•using advanced search options to limit and refine searches; and
•assessing whether an online resource or person is credible and trustworthy.
Furthermore, administrators have no tool to use to evaluate the digital literacy
competency of classroom teachers.
The remainder of the chapter will outline a summary of the current study, a
summary and interpretation of findings, implications, limitations of the study, and
recommendations.
Summary of Study
The purpose of the current study was twofold. The first purpose was to utilize a
panel of expert practitioners to identify the characteristics of a teacher with a high level of
digital literacy competency due to the lack of a widely agreed-upon definition. The
second purpose of the current study was to examine observable teacher behaviors integral
to teaching digital literacy in order to create a non-evaluative walk-through document for
measuring the digital literacy proficiency of teachers. Data from these walk-throughs may
be used by administrators or instructional technology staff to guide professional
development decisions.
Using a Delphi method study, expert educators in the digital literacy field
generated a list of observable teacher behaviors exemplifying a high level of competence
in digital literacy instruction. This information was formatted into a brief, non-evaluative
walk-through document. The current study was accomplished in four iterations, the first
of which identified the teacher behaviors, the second and third rounds focused on
achieving consensus on the items to be included on the definition and walk-through
document, and in the fourth round panelists provided input on the definition and
walkthrough document.
“In many schools, despite significant investment in technology, teachers are not
making effective use of the engaging instructional practices of digital and media literacy”
(Hobbs, 2010, p. 25). The reasons for the oversite may include teachers having a lack of
familiarity with technology tools or effective integration techniques. The current study
created a definition of a digitally literate teacher and a non-evaluative document
identifying possible areas of digital literacy deficiency in teachers and areas at which
teachers excel. The research questions the current study addressed are as follows:
Research Question 1 (RQ1) What are the indicators of digital literacy competency in
teachers as identified by a panel of experts?
Research Question 2 (RQ2) Which of these behaviors could be observed on a
walkthrough observation?
Research Question 3 (RQ3) What are the elements of such a form?
Suppo (2013) specifically addressed the need for “a tool to measure the level of
digital citizenship demonstrated daily by administrators, teachers, and students” (p. 96) as
a conclusion of his study. Hatlevik et al. (2014) recommend studies “to describe good
teacher roles for fostering students’ digital competence” (p. 229). The current study
addressed the gap in the literature as described in these two studies as well as those
identified by Jones and Mitchell (2015) and Blummer (2008).
Literature Review Overview
The literature review provided a review of relevant literature regarding digital
citizenship, literacies related to education in a digital environment, educational
technology standards, technology integration, classroom walk-throughs, Likert-type
scales, and the Delphi method. A brief overview of each of these headings is summarized
below.
Standards
The International Society for Technology in Education (ISTE) first published the
National Educational Technology Standards (NETS) in 1998. These standards included
basic operations and concepts: standards that addressed technology tools for productivity,
communications, research, and problem-solving and decision-making. Digital citizenship
and digital literacy were addressed in the standard for social, ethical, and human issues.
Corresponding standards for teachers were released in 1999.
Revised standards were released by ISTE in 2007. The expectations for digital
citizenship were expanded, and a new standard, research and information fluency was
added. Corresponding standards for educators was released in 2008.
In 2016, ISTE released updated standards named the ISTE Standards for Students.
The third version of standards included components for students to manage their digital
identity, demonstrate an understanding of intellectual property, and manage data to
promote digital privacy and security. The research and information fluency standard from
2007 was rewritten in 2016 under the heading “knowledge constructor.” Corresponding
standards were released by ISTE for educators in 2017.
Digital Citizenship
Ribble and Bailey (2004) first coined the term digital citizenship and defined it as
“the norms of appropriate, responsible behavior with regard to technology use” (p. 1). It
is currently the definition most commonly accepted in K-12 education and was the
definition used for the purposes of the current study.
Suppo (2013) recommended districts design a curriculum by first conducting a
digital citizenship audit to determine which resources are most needed. The purpose of
the current study, the creation of a walk-through document measuring the demonstrable
digital literacy proficiency of classroom teachers, could serve as a data source for the
digital citizenship curriculum audit Suppo recommends.
Literacies
There are several literacies that correlate with digital citizenship such as digital
literacy, information literacy, internet literacy, media literacy, and web literacy. These
literacies are indistinct and overlapping. Just as researchers have been unable to identify
an exact definition of digital citizenship, there is also a lack of agreement on the aspects
of these various literacies.
Digital literacy. Gilster (1997) first popularized the term digital literacy, defined
as “the ability to understand and use information in multiple formats from a wide range of
sources when it is presented via computers” (p. 1). The term also refers to the ability to
“access networked computer resources and use them” (Gilster, 1997, p. 1). Lastly, digital
literacy refers to the ability to solve problems using search methods and an understanding
of accessing information in a hypertext format. Finally, digital literacy includes “creating
an information cache that access the processing power of networked computers” (p. 33).
Ba et al. (2002) refined the definition for digital literacy as a “set of habits through
which youngsters use information technologies for learning, work, and fun” (p. 3). Digital
literacy also encompassed skills such as troubleshooting abilities, purposes for
connecting, and skills in using popular technology tools, communication literacy, and
web literacy (Ba et al., 2002).
Aviram and Eshet-Alkalai (2006) described digital literacy as a “survival skill in
the digital era” (p. 102). The pair noted research to date has been practice oriented and
lacked a theoretical foundation. Furthermore, Blummer (2008) stated there was a need for
more study on digital literacy. Blummer noted “the topic of digital literacy is
characterized by a nebulous definition and limited research studies that are often centered
in foreign countries” (p. 44). Nevertheless, it was “essential to correlate student digital
literacy habits in their personal lives to instructional practices at school” (Blummer, 2008,
p. 38).
The American Library Association (2013) highlighted the necessity for digital
literacy in order to function in a global, connected society in the report from the Office
for Information Technology Policy’s Digital Literacy Task Force. The American Library
Association Digital Literacy Task Force reported 80% of Fortune 500 companies posted
open positions online and required online applications and the majority of modern era
jobs required some level of technology skills. The organization defined digital literacy as
“the ability to use information and communication technologies to find, understand,
evaluate, create and communicate digital information” (American Library Association,
2013, p. 1).
Information literacy. Information literacy was originally coined in 1974 by
describing people who work regularly with information, such as librarians, as being
“information literate” by Zurkowski, the president at the time of the Information Industry
Association (p. 6). Information literate individuals “have learned technique and skills for
utilizing the wide range of information tools as well as primary sources in molding
information solutions to their problems” (Zurkowski, 1974, p. 6).
Koltay (2011) described information literacy, which predated digital literacy, as
“the process of recognizing information need, finding, evaluating and using information
to acquire or extend knowledge” (p. 33). Information literacy education prioritized
critical thinking, especially in an age where “gatekeepers” such as editors and reviewers
were not present online the same way they were in printed materials. This necessitated the
consumer of information also become the gatekeeper of said information.
Miller and Bartlett (2012) made the connection between digital literacy and
information literacy in that digital literacy combines “old techniques” such as critical
engagement with information “with new and specific knowledge bases about how the
internet works, and how, given how it works, it can inadvertently deceive or be
deliberately used to deceive” (p. 39).
Alvarez and Gisbert (2015) noted teacher participants “present major deficiencies
when it comes to evaluating the information they find” (p. 192). Teachers also
demonstrated difficulty determining the credibility of information on the internet “as
reliable and true in comparison with information they obtain from analogical sources
(only 52.4% do)” (Alvarez & Gisbert, 2015, p. 192).
Internet and web literacy. Johnson (2007) discussed various internet literacy
definitions as “the capability to access and evaluate online information,” online search
competence, and “skill with connectivity, security, communication, multimedia, and web
page development” (p. 434)
“Functional Internet literacy, which requires complex cognitive processing, is best
achieved in structured and directed learning situations. Formal teaching of functional
Internet literacy is based upon the assessment of cognitive skill deficits and instruction
that targets those identified deficits” (Johnson, 2007, p. 438).
November (2012) defined web literacy as “accurate and creative searching
techniques that are applicable across every discipline” (p. 50). “Web literacy isn’t just
about learning to be productive online; it also involves taking control of the technologies
that we use, so that we—not our tools—are guiding our results” (p. 72).
A lack of web literacy skills is problematic because students are “using paper
literacy skills to navigate in a digital world, and coming up with misleading and shallow
results” (November, 2012, p. 50). Students are often “web-illiterate” despite daily use of
technology and the Internet. Students must be educated, according to November, on how
to assess the origin of information and how to validate that information. Students must
also learn how to “research, publish, and communicate through and with the Internet and
other information tools” (p. 52).
Technology Integration
Farmer (2011) provided a partial roadmap for how to evaluate digital information
by engaging students in “active examination, debate, and self-reflection, educators can
use a variety of technological tools: threaded discussion, online chat, blogs, wikis, and
online conferencing” (p. 390), yet the current task still proves daunting to many
educators, especially those who first encountered the Internet as adult learners.
Teaching in the redefined role must include “techniques for applying knowledge
to produce information and facilitate communication…evaluating the resources they
decide to use” (November, 2012, p. 52). Likewise, students must be taught to examine all
information found on the Internet for purpose, author, and place and how the information
found on a particular website is linked to other information found on other websites. The
two most common models of technology integration, with the exception of the ISTE
standards, are described below. They are the Substitution, Augmentation, Modification,
Redefinition (SAMR) model and the Technological Pedagogical Content Knowledge
(TPACK) model.
SAMR model. The SAMR model was developed by Dr. Ruben R. Puentedura in
2006. The framework is a four-stage continuum or progression of technology integration.
The stages as defined by Puentedua (2006) are as follows:
•Substitution: Technology acts as a direct tool substitute with no functional
change.
•Augmentation: Technology acts as a direct tool substitute with functional
improvement.
•Modification: Technology allows for significant task redesign.
•Redefinition: Technology allows for the creation of new tasks previously
inconceivable (Puentedura, 2006).
TPACK model. The TPACK model originated as the Pedagogical Content
Knowledge (PCK) model by Shulman (1986). PCK described forms of knowledge
teachers needed to possess in addition to an understanding of subject matter and
curriculum. The technology component was added in 2006 by Mishra and Koehler to
further teacher integration of technology into pedagogy. Combinations of knowledge
types include:
•Content Knowledge: subject matter being taught;
•Pedagogical Knowledge: process and methods of teaching;
•Technological Knowledge: technologies such as the internet and computers;
•Pedagogical Content Knowledge: pedagogy applicable to teaching specific
content;
•Technological Content Knowledge: manner in which technology and content
are related; and
•Technological Pedagogical Knowledge: capabilities of technologies as they
can be used in various teaching methods and how teaching changes as a result
of technology.
Walk-through
A walk-through was defined by The Center for Comprehensive School Reform
and Improvement (2007) as a “brief, structured, nonevaluative classroom observation by
the principal that is followed by a conversation between the principal and the teacher
about what was observed” (p. 1). A walk-through occurred in a shorter time span than that
of a formal observation yet because it was performed more often, it allowed an
administrator to see more classroom instruction over time than a one-time formal
observation. Frequent, consistent walk-throughs enabled administrators to observe
patterns of instruction. The primary purpose of a walk-through is “to provide a structure
for dialogue between principal and teacher about what goes on in the classroom” (The
Center for Comprehensive School Reform and Improvement, 2007, p. 2).
The Center for Comprehensive School Reform and Improvement (2007) has
outlined several essential elements of a walk-through. They include brevity, focus, and
dialogue. Walk-throughs must be brief in order to allow the administrator to maximize the
number of classrooms in a given time period. Teachers should be given a purpose for the
walk-through in order for it to be most effective. Lastly, an effective walk-through is a
tool to facilitate feedback and reflection among the teacher and administrator.
Kachur et al. (2010) conducted an examination of 18 models of classroom
walkthroughs for components such as purposes, participants, teacher involvement,
focuses or look-fors, protocols, data recording, and follow up. Kachur et al. (2010)
defined walkthroughs as “short, informal observations of classroom teachers and students
by school administrators, coaches, mentors, peers, and others, followed by feedback,
conversation, and/or action” (p. 1). Classroom walk-throughs served to create a snapshot
of instruction and student learning that can identify strengths, patterns, and needs of an
organization. The recommendations for look-fors and document construction were used
during the creation of the walk-through document created for the current study.
Likert-type Scales
Likert scales were named for R. Likert, who initially published a scale for rating
participant attitudes. “On this basis one of our cardinal problems is to find whether social
attitudes, in this sense, can be shown to be measurable, and if an affirmative answer is
forthcoming, a serious attempt must be made to justify the separation of one attitude from
others” (Likert, 1932, p. 8). Likert noted attitudes could be “clustered” and placed on a
continuum which allows for a numeric value to be assigned. Thus, participant attitudes
about a social science topic may be measured.
Likert scales have been generally accepted to be ordinal as opposed to interval.
This is due to the fact that while the response choices on a Likert-type scale have an
order, the differences, or intervals, between those points may not be equal (Jamieson,
2004). Researcher recommendations as to appropriate statistical measures and the
inclusion of a 5-point Likert-type scale were utilized in the current study.
Delphi Method
The current study utilized the Delphi method. The Delphi technique was named
for the ancient Greek oracle at Delphi who “offered visitations of the future for those
seeking advice” (Hanafin, 2004, p. 4). The oracle was one who has “unquestioned
wisdom and knowledge or of infallible authority” (Yousuf, 2007, p. 1). The Delphi
method has roots in the philosophy of educator John Dewey who believed that social
science research should directly impact real-world practice and decision-making (Brady,
2015).
The Delphi method was first utilized by the Rand Corporation in the 1950’s as
Project DELPHI. The first applications of the Delphi method were “to assess the direction
of long-range trends, with special emphasis on science and technology, and their probable
effects on society” (Gunaydin, n.d., p. 2). The classical Delphi technique study contains
five features: “anonymity, iteration, controlled feedback, statistical group response and
stability in responses among those with expertise on a specific issue” (Hanafin, 2004, p.
5). The purpose of the arrangement is to allow the experts to reach consensus on a
particular topic.
The controlled method of questioning in several iterations appeared to be “more
conducive to independent thought on the part of the experts and to aid them in the gradual
formation of a considered opinion” (Dalkey & Helmer, 1963, p. 459). Revisiting the
problem multiple times and examining answers provided by the group allowed each
expert to correct misconceptions and draw conclusions about various aspects of the
problem that he might not have considered in the initial round of questioning. It was
specifically designed for “participants to reassess their initial judgments about the
information provided in previous iterations” (Hsu & Sandford, 2007, p. 2).
Methodology
The Delphi method has been popular among educational researchers due to the
availability of current practitioners to serve as experts, as demonstrated by the four
studies outlined in the review of literature contained in this chapter. Recommendations
for conducting successful Delphi method research were followed during the process of
the current study.
The Delphi method is particularly useful in the following situations:
•The problem does not lend itself to analytical techniques but can benefit from
“subjective judgements on a collective basis” (Hanafin, 2004, p. 9.)
•The experts on a particular subject are “In different fields and occupations and
not in direct communication” (Hanafin, 2004, p. 9).
•“The number of specialists is too large to effectively interact in a face-to-face
exchange and too little time and/or funds are available to organize group
meetings” (Hanafin, 2004, p. 9).
There are four requirements for expertise in a Delphi study: a) knowledge and
experience with the issues under investigation; b) capacity and willingness to participate;
c) sufficient time to participate in the Delphi; and, d) effective communication skills
(Skulmoski et al., 2007).
Expert Panelists
Experts in the fields of instructional technology, school administration,
professional development, and school librarianship were polled for participation. An
expert for the purposes of the current study was defined as an educator who has worked
in his/her field for at least 10 years and has demonstrated competence by publishing,
presenting, or actively participating in professional organizations at the state, national, or
international level on the topics of web literacy, information literacy, digital literacy,
professional learning for instructional technology implementation, and/or digital
citizenship.
The expert panelists represented nine states and three countries. Ten of the
participants were from Texas, followed by three in Maryland, and three from
Massachusetts. Two panelists were in Illinois. One panelist each represented Alabama,
Kansas, Kentucky, Michigan, New York, and Pennsylvania. One panelist was in Cairo,
Egypt and one was in Singapore. This wide geographical area of experts was noted by
Brady (2015) as a distinct advantage to using the Delphi method.
Summary of Findings and Interpretation of Results
The Delphi technique typically contains at least three rounds of questionnaires to
generate consensus or a sufficient statistical analysis of responses. The current study
utilized four rounds. The first round identified the teacher behaviors, the second and third
consisted of ratings using a Likert-type scale to determine which items most needed to be
in the definition and on the walk-through document, and the fourth round provided an
opportunity for participants to provide input on a sample definition and walk-through
document. The creation of the definition and walk-through document matched the
practical purpose of a Delphi study to guide practice, policy, or decision making as
identified by Brady (2015).
Round 1
Members of the expert panel were provided a link to a survey instrument
including an informed consent option, questions about demographic information, and
three open ended questions. These can be found in Appendix A. The open-ended
questions were as follows:
Delphi Question 1 (DQ1) What qualities does a teacher possess and demonstrate that
indicate a high proficiency in digital and web literacy?
Delphi Question 2 (DQ2) What are the observable behaviors of a teacher with a high
level of digital literacy competence?
Delphi Question 3 (DQ3) What are the observable behaviors of students in a classroom
with a teacher demonstrating a high level of digital literacy competence?
The responses were sorted and separated into descriptive words or phrases due to
the fact that many panelists answered the question in a paragraph format. Exact wording
was preserved as much as possible. There was a total of 173 distinct items identified by
panelists overall in response to the three open ended questions.
Five themes emerged from the panelist responses to DQ1. The themes were
characteristics, technology integration, digital and/or information literacy, pedagogy, and
miscellaneous. The 55 items were sorted by theme and alphabetized in preparation for
panelists to rate using a Likert-type scale in Round 2.
Four themes emerged from the panelist responses to DQ2. The themes were
characteristics, technology integration, digital and/or information literacy, and pedagogy.
The 57 items were sorted by theme and alphabetized in preparation for panelists to rate
using a Likert-type scale in subsequent rounds.
Four themes emerged from the panelist responses to DQ3. The themes were
characteristics, technology integration, digital and/or information literacy, and
instructional activities. The 61 items were sorted by theme and alphabetized in
preparation for panelists to rate using a Likert-type scale during subsequent rounds.
Round 2
The second round of the current study presented panelists completing Round 1
with an online survey instrument created from the responses received in the first round,
which can be found in Appendix B. The survey included a list of observable behaviors
obtained from the first round that panelists were asked to rank using a Likert-type scale of
1-5 as recommended by Nadler et al. (2015). The round closed with 19 expert panelists
responding out of the original 26 panelists from Round 1. This was a 73% response rate.
Responses at the close of Round 2 were entered into SPSS software. Statistical
analysis for mean, median, mode, and standard deviation was calculated to determine
consensus for each item. A Cronbach’s alpha statistical analysis of reliability and internal
consistency was also conducted on the 173 items at the conclusion of Round 2.
Cronbach’s alpha showed the questionnaire to reach acceptable reliability, α = 0.988. The
closer Cronbach’s alpha coefficient is to 1.0 the greater the internal consistency of the
items in the scale.” (Gliem & Gliem, 2003, p. 87).
Of the original 55 items for DQ1, 31 items reached consensus for inclusion in the
definition of a digitally literate teacher, 12 items reached consensus for elimination from
the definition, and 12 items failed to reach consensus. Of the original 57 items for DQ2,
33 items reached consensus for inclusion in the walk-through document, 12 items reached
consensus for elimination from the document, and 11 items failed to reach consensus. For
DQ3, 27 of the original 61 items reached consensus for inclusion in the walk-through
document, 15 items reached consensus for elimination from the document, and 19 items
failed to reach consensus.
Round 3
At the conclusion of Round 2, 42 items from the three Delphi questions failed to
reach consensus for inclusion in the definition or the walk-through document. An online
survey instrument, found in Appendix D, was created of these items asking panelists to
rate them again using a Likert-type scale.
At the conclusion of Round 3, the items were analyzed for central tendencies of
mean, median, mode, and standard deviation using SPSS software. A Cronbach’s alpha
statistical analysis was performed to determine reliability and internal consistency of the
Likert-type scale provided to expert panelists during Round 3. Cronbach’s alpha showed
the questionnaire to reach acceptable reliability, α = 0.978.
Of the 42 items presented to the expert panelists, 10 items reached consensus to
be included in the definition or walk-through document, 13 items reached consensus for
elimination from the definition and the document, and 19 items failed to reach consensus.
Items failing to reach consensus after three rounds of the current study were eliminated
from consideration for the definition and the walk-through document to prevent panelist
fatigue. It has been recommended that two or three rounds of a Delphi study are sufficient
for this reason (Hsu & Sandford, 2007; Verbeke, 2014). The current study was completed
in three iterations or four rounds.
Round 4
When consensus had been reached on the essential elements for a definition and
the look-fors for the walk-through form, the mock-up of the definition and the form was
sent to panelists during Round 4. A copy of each of these can be found in Appendices F
and G. Of the 26 panelists receiving the email invitation to complete the online survey, 21
panelists rated the definition of a digitally literate teacher, and 20 completed the entire
Round 4, which was a 77% response rate.
Panelists were asked to review the definition and select one of the following
options:
•Accept the definition of a digitally literate teacher as written; • Do not accept
the definition of a digitally literate teacher as written; or
•Accept the definition of a digitally literate teacher with modification.
Panelists selecting the third option were requested to provide the suggested
modification in the comments field following the question.
Of the three options, 95.27% of the panelists opted to approve the definition,
either as written or with modifications. Fourteen of the panelists, or 66.7%, selected the
first option, to accept the definition of a digitally literate teacher as presented. Another
28.57%, or six panelists, selected to accept the definition with modifications and one
panelist chose to reject the definition.
The document presented to expert panelists in Round 4 attempted to address RQ3,
which was, “What are the elements of such a form?” A link to the Proposed Stout Digital
Literacy Walk-through document was presented to panelists. The document was stored in a
secure online Google Drive environment and was included as a link in the fourth round of the
panelist survey. A sopy of this document can be found in Appendix G.
Of the 26 recipients of the document, 20 panelists responded. The vast majority of
panelists, 85%, elected to accept the proposed walk-through document either as written or
with modification. Half of the panelists accepted the document as written, seven (35%)
accepted the document with modification. Three panelists (15%) did not accept the
walkthrough document.
An additional question posed to panelists in Round 4 was “Who do you think
would use this document? Please select all that apply.” Options presented to panelists
included:
•Campus principal or assistant principal;
•Central office administrator;
•Instructional coach, specialist, facilitator; or
•Other (please specify).
The largest number of respondents selected instructional coach, specialist,
facilitator (19), followed by campus principal or assistant principal and central office
administrator, both of which received 15 votes. Six panelists selected the “other” option.
Three of the panelists recommended teachers. Technology vendors, teachers, curriculum
coordinators, parents, and students were each mentioned once.
Twenty of the 21 participants accepted the definition of a digitally literate teacher
either as written or with modifications. Seventeen of 20 panelists accepted the
walkthrough document either as written or with modifications. Each of the panelists
indicated at least one possible group that would be able to make use of the walk-through
document, the most highly selected being an instructional coach, specialist, or facilitator.
Discussion of Findings by Research Question
Research Question 1
RQ1 was, “What are the indicators of digital literacy competency in teachers as
identified by a panel of experts?” Expert panelists were presented with DQ1, which was,
“What qualities does a teacher possess and demonstrate that indicate a high proficiency in
digital and web literacy?” At the conclusion of four rounds, the following indicators of a
digitally literate teacher were approved by 95% of the expert panelists:
A digitally literate teacher is one who demonstrates proficiency in technology
integration, digital/information literacy skills, and pedagogy in an online environment.
The digitally literate teacher possesses the following characteristics:
•the ability and willingness to search for answers to questions;
•ethics;
•persistence when implementing technology; grit;
•problem-solving skills;
•risk-taking; and
•a willingness to fail.
In regard to technology integration, the digitally literate teacher:
•is fluent with technology–displays a positive attitude and affinity towards
technology, is able to explain technology to others, explores and troubleshoots
technology-related issues;
•frequently uses technology and digital resources in meaningful ways;
appropriately embeds technology into curriculum as a tool to enhance
learning;
•provides students with personalized learning, choice, and ownership through
technology;
•promotes digital equity to prevent educational gaps among students; and
•models leadership and appropriate use of technology for students and
colleagues.
The digitally literate teacher demonstrates an understanding and commitment to
the principles of digital citizenship, digital literacy, and information literacy by:
●experimenting with and learning about new web/digital literacies;
●focusing on high level, critical thinking while instructing students in digital
citizenship, digital and web literacy, research skills, and copyright law; and
●locating, evaluating, synthesizing, and using information effectively and
efficiently.
A digitally literate teacher demonstrates an understanding of technology-related
pedagogy through:
●assuming varied roles in the learning process with digital age learners such as
that of facilitator, learner, supporter, and challenger;
●practicing current instructional technology strategies and best practices;
●combining knowledge of subject matter and integration of technology tools in
a manner beneficial to all learners; and
●integrating ISTE and state technology standards into instruction.
Research Question 2
RQ2 was, “Which of these behaviors could be observed on a walk-through
observation?” Expert panelists were presented with two questions about observable
behaviors; one question about teacher behaviors and one question about student
behaviors. These questions were DQ2 and DQ3.
Delphi Question 2 (DQ2) What are the observable behaviors of a teacher with a high
level of digital literacy competence?
At the conclusion of Round 3, panelists identified and reached consensus on 33
observable behaviors for the teacher as seen in Appendix E. These behaviors were
grouped under four themes: characteristics, technology integration, digital and/or
information literacy, and pedagogy.
Delphi Question 3 (DQ3) What are the observable behaviors of students in a classroom
with a teacher demonstrating a high level of digital literacy competence?
At the conclusion of Round 3, panelists identified and reached consensus on 33
observable behaviors for students. These behaviors were grouped under four themes:
characteristics, technology integration, digital and/or information literacy, and
instructional activities.
Research Question 3
RQ3 was, “What are the elements of such a form?” Using the 33 items identified
by expert panelists for teacher behaviors and the 33 items identified for student behaviors,
a proposed walk-through document was created and presented to panelists for approval.
Construction of the document relied upon recommendations made by researchers
found during the literature review process. Gillespie (2016) noted walk-throughs require a
well-designed rubric to facilitate brevity and efficiency. Walk-through expectations
should be clear and objective, which also require a well-designed rubric with observable
look-fors. Look-fors, according to Kachur et al. (2010) are “clear statements or
descriptors of observable evidence of teaching and learning such as specific instructional
activities, learning activities, behavioral outcomes, artifacts, routines or practices” (p. 76).
For the purposes of the current study, look-fors focused on the level of digital literacy
practices exhibited by teachers and students in a classroom.
The format of the document was described by Kachur et al. (2010) as a
combination form. The combination form included both a checklist and space for notes
and was utilized to assist “managing productive conversations.” (Kachur et al., 2010, p.
99). The checklist portion of the form was chosen in compliance with recommendations
by Kachur et al. (2010). A checklist was utilized when there were “specific look-fors that
can be easily observed” (p. 91). The checklist data recording method clarified “exact
educational behaviors and activities expected” (Kachur et al., 2010, p. 92). The contents
of the proposed walk-through document are found in Figures 11 and 12.
Implications
The current study had two purposes, the first of which was to address a gap in the
literature regarding the definition of a digitally literate teacher as identified by Blummer
(2008). Blummer (2008) noted “the topic of digital literacy is characterized by a nebulous
definition and limited research studies that are often centered in foreign countries” (p.
44). The definition of a digitally literate teacher at the conclusion of the current study
may guide further research in the topic.
Eshet-Alkalai (2004) developed a skills framework as a definition of digital
literacy. The framework includes:
•Photo-visual skills (gleaning information from graphical displays);
•Reproduction skills (creating new materials from preexisting ones);
•Branching skills (gathering knowledge from website navigation);
•Information skills (evaluating quality and validity of information); and
•Socio-emotional skills (understanding and utilizing “rules” of online
communication).
Hobbs (2010) also defined digital literacy as a set of skills including:
•making responsible choices; accessing and comprehending information;
analyzing messages in a variety of forms;
•creating content in a variety of forms;
•reflecting on conduct and communication behavior; and
•taking social action by working individually and collaboratively to share
knowledge and solve problems.
While both of these skillsets are valuable, not all aspects of these can be observed
in a classroom walk-through. In addition, these definitions pertain to behavior of an
individual but do not necessarily correspond to that of a classroom teacher.
The definition of a digitally literate teacher produced as a result of the current
study describes aspects that are observable during a classroom walk-through. These
aspects include a demonstration of proficiency in technology integration, digital/
information literacy skills, and pedagogy in an online environment.
Stout Digital Literacy Walk-through Document
A digitally literate teacher is one who demonstrates proficiency in technology integration,
digital/ information literacy skills, and pedagogy in an online environment.
Students in a digitally literate teacher’s class are empowered and engaged learners who
effectively use technology to answer questions, solve problems, and demonstrate
knowledge.
Teacher and student behaviors are organized into three domains: technology integration,
digital/information literacy, and pedagogy/instructional activities. This organization
allows for a focused observation of one domain or a holistic observation of all three
domains.
Technology Integration
The teacher: The students: Notes/Observations
• Effectively integrates technology
for critical thinking, creativity and
productivity
• Introduces technology-related
current events when appropriate to
demonstrate the impact of
technology on everyday life
• Encourages student choice and
exploration in technology
applications
• Differentiates content, process, and
products using technology
• Shares experiences and digital
resources with colleagues
• Uses varied resources in whole
class, small group and individual
student settings in engaging and
meaningful ways
•
•
•
•
•
•
•
•
Locate resources to find and use
information efficiently
Teach others how to use tools and
troubleshoot
Choose appropriate resources to
create products demonstrating
learning
Effectively use technology to
create, organize, and access
safely curated materials
Share knowledge in multiple
ways Speak, listen, read, write,
and present with technology
Store, use, and create content
online
Use a variety of digital tools
appropriate for products they
create
• Experiment with new technologies
Figure11. First page of proposed “Stout Digital Literacy Walk-through Document” as
presented to expert panelists during Delphi Round 4.
Digital/Information Literacy
The teacher: The students: Notes/Observations
• Demonstrates copyright, digital
citizenship, and research skills
• Models online communication and
computational thinking skills
• Appropriately uses social media and
global communication tools in class
• Finds teachable moments to model
legal and ethical behaviors and
expectations associated with online
access
• Verifies Terms of Service and use of
student data for digital resources
• Familiarizes self with federal
mandates, district AUP, and other
policies regarding safe and effective
• Comply with federal copyright laws
and district policies such as the AUP
• Practice digital citizenship at all
times
• Demonstrate appropriate online
behavior, respect for intellectual
property, and appropriately credit
others for their work
• Show a consideration for digital
privacy
• Work with peers, teachers, and
parents to obtain digital skills
use of digital resources
• Integrates digital literacies in
lessons
• Credits image/video/music sources
and requires the same from students
Pedagogy/ Instructional Activities
The teacher: The students: Notes/Observations
• Uses a variety of teaching strategies
to engage students at all
developmental levels
• Designs high quality assessments to
check for understanding
• Acts as facilitator, providing
students access to tools and
resources in a manner that lets
students be the experts
• Allows students to learn from
mistakes and choose how to
synthesize learning
• Uses technology to provide tiered,
personalized instruction and
scaffolding
• Continually learns new strategies
and tools; models their
implementation
• Communicate what they are doing
• Are engaged in activities that
require high levels of thinking and
encourage collaboration
• Apply inquiry-based learning
• Understand different
registers/purposes for digital
communication
• Use technology and critical thinking
to solve real-world problems and
demonstrate learning
• Work within clear academic
objectives and behavioral norms set
by the teacher
Figure 12. Second page of proposed “Stout Digital Literacy Walk-through Document” as
presented to expert panelists during Round 4.
The second purpose of the current study was to ascertain observable behaviors of
a classroom taught by a teacher with a high level of digital competency in order to create
a walk-through document of these behaviors. The walk-through document attempted to
address recommendations for further research as identified by Blummer (2008), who
asserted much of the research on digital literacy focuses on student behaviors and skills
acquisition (Blummer, 2008). Suppo (2013) specifically addressed the need for “a tool to
measure the level of digital citizenship demonstrated daily by administrators, teachers,
and students” (p. 96) as a conclusion of his study. Hobbs (2010) recommended
developing an assessment of digital and media literacy competence for teachers. Hatlevik
et al. (2014) recommended studies “to describe good teacher roles for fostering students’
digital competence” (p. 229).
Prior to the completion of the current study, a document to assess the digital
literacy competence of classroom teachers was not available. The document, after
revision and field testing, may be utilized nation-wide to provide benefits outlined by
Kachur et al. (2010) pertaining to walk-throughs. These benefits include “appraising how
professional development initiatives are being incorporated into classroom practices” and
“identifying professional development needs of the faculty and staff” (p. 8).
Limitations
Panelists were selected from a convenience sample of experts; corresponding data
may have been limited due to the number of participants and the method of selection.
Panelists also offered conflicting opinions throughout the study when rating individual
items. Items reaching consensus for inclusion also included negative panelist comments.
A number of these comments suggested some panelists failed to consider “observable”
and “limited to digital literacy” when rating items.
Panelist comments also reflected conflicting opinions and possible agendas by
expert panelists. For example, items related to the SAMR model received comments
throughout the process by panelists both for and against inclusion. Sample comments
leading to the two items related to the SAMR model being eliminated included:
•Is SAMR the best model? There is a bit of controversy over this.
•Not demonstrable
•“Not actionable”
•“Again, this is a doctoral dissertation. While my own dissertation is absolutely
BEDAZZLING, knowledge of mine is not required to be a good teacher with
technology”
•“Again, I question the validity of the SAMR model. Perhaps TIMS is better?”
• “SAMR is NOT research based”
However, when the definition and walk-through document were presented to
panelists during Round 4, two comments were made regarding its exclusion: “I suggest
that a digitally literate teacher is familiar with models such as SAMR and TPACK to
understand the cognitive demand of tasks and appropriate usage of digital tools” and
“Showing all items in a rank format would be more helpful. For example, SAMR model
would be very helpful observable behavior, but it is off the list. It may have a lower
ranking, but it shouldn’t be eliminated.” These comments suggest at least one panelist
held strong opinions regarding the SAMR model being eliminated from contention.
The current study was also limited to observable behaviors of classroom teachers
and librarians conducting direct, face-to-face instruction. Due to time limitations and the
scope of the project, the current study was limited to the K-12 educational environment
and may or may not have applications to university students and professors.
Lastly, the current study was limited in that it did not address the effectiveness of
digital literacy instruction. The current study also did not evaluate current digital
citizenship or digital literacy curricula. Nor did the current study determine whether or
not instruction has a lasting impact on student digital behaviors or student achievement.
Recommendations
Recommendations for further research apply primarily to the walk-through
document produced as a result of the current study. The next logical step for the
document would be to field test the instrument. This will occur in two school districts
during the 2018-2019 school year. One of the districts, a large suburban district in north
Texas has indicated interest to conduct a pilot of the form at during the next school year.
At the conclusion of the current study, the researcher was contacted by an administrator
in a second district, a mid-sized district in south Texas about field testing the instrument.
Prior to field testing, the document itself must be evaluated in the context of panelist
recommendations. Possible adjustments based on panelist comments include:
•evaluating that each individual item would be observable during a brief
classroom visit;
•rearranging items to ensure similar items are grouped together; and
•moving the Pedagogy component to the beginning of the document.
Other considerations that will need to be made prior to pilot testing include
creating a training component for staff using the document and teachers that will be
observed. Writing instructions for use and adding a scoring rubric would also need to be
considered.
One option of using the document that may be beneficial to educators would be to
only observe one component during a “mini-walk-through.” This option would need to be
reflected in any instructions, professional development, or rubrics created.
Another recommendation is to create resources related to the observable behaviors
such as examples of the behaviors and professional development opportunities or
resources aligned to the behaviors that administrators can recommend to teachers.
The last recommendation prior to field testing would be to develop an
implementation plan that includes the following elements:
•timeline for professional development and data gathering;
•control groups if any are to be used;
•data collecting procedures; and
•method of soliciting participant feedback.
Further revisions to the document and implementation process based on results of
field testing would be almost certain.
Conclusions
Technology skills are crucial for students in order to function effectively posthigh-
school in an increasingly digital world (Freeman et al., 2017; National Academy of
Sciences, 2017; Rainie & Anderson, 2017). Research has demonstrated students are
generally not able to perform at high levels of digital literacy competence (Boyle, 2010;
Lee 2014; Stanford History Education Group, 2016). Research has also proven these
skills for the most part cannot be self-taught (Johnson, 2007; Li & Ranieri, 2010; Lindsay
& Davis, 2010; November, 2012).
It would be logical to assume teachers have the responsibility to guide students to
higher-level digital literacy skills. Yet research has demonstrated most teachers do not
have a high level of digital literacy and technology integration proficiency either
(Alexander et al., 2016; Alvarez & Gisbert, 2015; Berardi, 2015; Downy & Donovan,
2011; Hobbs, 2010; Jones & Mitchell, 2015; Miller & Bartlett, 2012; Patton, 2015;
Pfaffe, 2017).
To address this problem, districts will need to prioritize high-quality, ongoing
professional development in digital literacy skills instruction and practice-based
technology integration methods. The definition of a digitally literate teacher created as a
result of the current study and the Stout Walk-through Document could serve as resources
to guide professional development decisions. In addition, educators will need to develop
a sense of urgency about honing their craft to include digital literacy skills practice at all
grade levels in every content area. Administrators must monitor and prioritize the
integration of digital literacy skills into instruction. To do any less would be a disservice
to students.
