Week 3 Assignment 1.5-3 pages

244

Article

Management of a Large Qualitative Data Set: Establishing Trustworthiness of the Data

Debbie Elizabeth White, RN, PhD

Associate Professor, Associate Dean of Research

Faculty of Nursing

University of Calgary

Calgary, Alberta, Canada

Nelly D. Oelke, RN, PhD

Assistant Professor

School of Nursing

Faculty of Health and Social Development

University of British Columbia, Okanagan Campus

Kelowna, British Columbia, Canada

Steven Friesen, BSc

Quality Practice Leader

Bethany Care Society

Calgary, Alberta, Canada

© 2012 White, Oelke, and Friesen.

Abstract

Health services research is multifaceted and impacted by the multiple contexts and

stakeholders involved. Hence, large data sets are necessary to fully understand the complex

phenomena (e.g., scope of nursing practice) being studied. The management of these large

data sets can lead to numerous challenges in establishing trustworthiness of the study. This

article reports on strategies utilized in data collection and analysis of a large qualitative study

to establish trustworthiness. Specific strategies undertaken by the research team included

training of interviewers and coders, variation in participant recruitment, consistency in data

collection, completion of data cleaning, development of a conceptual framework for analysis,

consistency in coding through regular communication and meetings between coders and key

research team members, use of N6 TM

software to organize data, and creation of a

comprehensive audit trail with internal and external audits. Finally, we make eight

recommendations that will help ensure rigour for studies with large qualitative data sets:

organization of the study by a single person; thorough documentation of the data collection

and analysis process; attention to timelines; the use of an iterative process for data collection

and analysis; internal and external audits; regular communication among the research team;

adequate resources for timely completion; and time for reflection and diversion. Following

these steps will enable researchers to complete a rigorous, qualitative research study when

faced with large data sets to answer complex health services research questions.

245

Keywords: qualitative research, data management, large data sets, rigour, nursing, scope of

practice

Acknowledgements: We want to thank all the participants for their valuable contributions to

this study. In addition, we would like to thank our funders: Canadian Health Services

Research Foundation, Calgary Health Region (now Alberta Health Services, Calgary),

Capital Health Region (now Alberta Health Services, Edmonton), Saskatoon Health Region,

and the University of Calgary, Faculty of Nursing.

International Journal of Qualitative Methods 2012, 11(3)

246

Introduction

Healthcare systems are very complex, multi component systems that are continually evolving.

Given the complexity of these systems, health services research is multifaceted and impacted by

the multiple contexts and stakeholders involved. Even when researchers study only a particular

component of the healthcare system (e.g., scope of nursing practice in acute care), multiple

contexts are encountered and many participants are included to better understand the complex

phenomena being studied. Health services research on scope of practice is not well established

and, given the research questions, qualitative research is often the focus of such exploratory

research. Because data collection may occur across a number of sites by more than one research

assistant, research teams encounter logistical issues and have difficulties maintaining

predetermined timelines. The end result can be a very large amount of qualitative data that must

be analyzed and interpreted appropriately to ensure that an accurate synopsis of the results will be

presented. Organization of data and attention to rigour are essential when working with such large

qualitative data sets. This article describes the management of a large qualitative data set

generated from the research study entitled “A Systematic Approach to Maximizing Nurses’ Scope

of Practice.” More specifically, the purpose of this article is to reflect upon and describe the

processes through which the research team managed a large qualitative data set to ensure that the

final product would be judged as rigorous. One of the authors, Nelly D. Oelke, has considerable

experience in qualitative research. This expertise and the application of the literature on

qualitative data analysis guided the structures and processes used to collect and analyze our data.

This article contributes to the literature about managing large qualitative data sets by providing

concrete steps for ensuring rigour in data collection, analysis, and interpretation.

Background

Trustworthiness and data management are vital to the success of qualitative studies. Although

literature on maintaining rigour in qualitative research is abundant, few articles have tackled

doing so with large qualitative data sets (Knafl & Ayres, 1996) and few researchers have

documented their process. A search of the literature was conducted and confirmed these findings.

Search terms used for our literature search included qualitative research, data management, large

data sets, and rigor, with coverage of the following databases: MEDLINE, CINAHL, and

PsycINFO. Guba (1981) and others (Johnson & Waterfield, 2004; Whittemore, Chase, & Mandle,

2001) recommend general methodologies to ensure rigour in qualitative research. Although the

descriptions of methodologies in the literature vary, most involve steps to maintain credibility,

dependability, transferability, and confirmability (Guba, 1981).

To maintain credibility (Guba, 1981) or authenticity (Whittemore et al., 2001), researchers must

adhere to methods accepted as scientifically sound in the qualitative and informational sciences.

While transparency of methodology is important, Sandelowski (1997) cautions against focusing

only on methods. Rather, researchers should maximize data utility to answer the research

questions. The researcher must have a satisfactory cultural familiarity with the participating

institution and use a comfortable approach in recruiting participants so that the sampling process

is random and unbiased (Guba, 1981). Moreover, participants’ input must be honest, clearly

recorded, and accurately presented (Whittemore et al., 2001).

Dependability and transferability are related in that both ensure all research design and operations

are clearly identified (Guba, 1981). These steps also allow for replication of the methodology

with a larger population or by future researchers. However, it is important to differentiate

between the dependability of a method in producing similar interpretations and the reliability of a

method in producing identical results. Qualitative research focuses on describing participants’

International Journal of Qualitative Methods 2012, 11(3)

247

experience as accurately as possible (Sandelowski, 1997), rather than using numbers to describe

the phenomena of interest. According to Sandelowski (1997), interpreting the results, providing

valid applications of the findings, and accumulating knowledge as a foundation for other studies

are essential for validating data from a qualitative study. Johnson and Waterfield (2004) explain:

Qualitative data are descriptive, unique to a particular context and therefore cannot be

reproduced time and again to demonstrate ‘reliability’(Bloor, 1997). Instead of trying to

control extraneous variables, qualitative research takes the view that reality is socially

constructed by each individual and should be interpreted rather than measured; that

understanding cannot be separated from context. (p. 122-123)

Whittemore et al.’s (2001) framework for enhancing rigour includes criticality and integrity

components, which for Guba (1981) and Johnson and Waterfield (2004), are included as

components of confirmability or an audit trail. It is recommended that researchers keep an

accurate, comprehensive record of the approaches and activities employed in the study, both in

data collection and analysis. This record includes highlighting shortcomings of the study in the

research report and providing transparent links between study results and actual experiences of

the participants in the study (Guba, 1981). Such audit trails not only provide a solid

methodological reference for the reader, but also provide an opportunity for reflective reasoning

(on themes or categories chosen, interpretations, etc.) and criticism for the researchers as the

study progresses (Guba, 1981; Johnson & Waterfield, 2004; Whittemore et al., 2001). For

example, if methodology changes at some point in the study, an audit trail would keep a record of

when, why, and what changes were implemented. Such audit trails become especially useful in

the management of large databases and for placing data points, methodology, and interpretation

within the particular context in which they belong.

Knafl and Ayres (1996) offer researchers two data management steps for handling larger

qualitative data sets. First, case summaries can save researchers great time and logistical

resources while decreasing error. Core study researchers would summarize focus group or

interview transcripts to a fraction of their original length, and also include relevant data organized

into themes agreed upon beforehand as part of a summary guideline. This step not only allows

core researchers, who will be interpreting the data, to work closely with the data, but also allows

for critical insight. As a complement to the case summaries, it is recommended that researchers

tackling a large data set create matrices using database management systems. Guided by themes

and questions identified in the study, matrices provide a visual display of the data, including

extracted themes. Such matrices simplify the data for researchers’ discussions, while the case

summaries provide more details on the data. Moreover, data can be reorganized quickly using

electronic matrices, allowing for various perspectives and discussions on study outcomes (Knafl

& Ayres, 1996).

Ensuring rigour in qualitative research is a priority when collecting, presenting, and interpreting

data. Larger qualitative data sets can present a critical challenge for researchers in maintaining

study trustworthiness and, therefore, special guidelines must be strictly followed to ensure

transparency, logical reasoning, and criticality. As few sources in the literature have suggested

methodology for managing large qualitative data sets, this article aims to outline the methods

followed by our research team to maintain rigour in such circumstances.

Description of the Study

Numerous reports have highlighted the need to address the under-utilization of health human

resources by maximizing professional scopes of practice (Advisory Committee on Health Human

International Journal of Qualitative Methods 2012, 11(3)

248

Resources, 2002; Fyke, 2001). The need to clarify and define the nursing scope of practice was

recognized in Canada as well as internationally. However, there was a void in the research

literature in terms of describing scope of practice (being able to practice to the full extent of one’s

education, knowledge, and experience) and examining barriers to the enactment of full scope of

practice. This research study was unique in that it examined scope and boundaries in the practice

of various categories of nursing personnel simultaneously, for example, registered nurses (RNs),

registered psychiatric nurses (RPNs), and licensed practical nurses (LPNs). The overall goal of

this research was to make rich and robust conclusions about the scope of practice of nurses, the

barriers to and facilitators of scope, and the impact of contextual factors on scope of practice.

Research findings have been reported elsewhere (Oelke, White, Besner, Doran, McGillis-Hall, &

Giovannetti, 2008; White, Oelke, Besner, Doran, McGillis-Hall, & Giovannetti, 2008).

Study Methodology

This research study used a descriptive exploratory design with mixed methods (Creswall, 2009)

to explain enactment of scope of practice among all categories of regulated nurses (e.g., RNs,

LPNs, RPNs). Both quantitative (e.g., questionnaires) and qualitative (e.g., interviews with a

variety of stakeholders) data were collected, and one informed the other in the data analysis and

interpretation. This article focuses on the qualitative data set. To make our experience of

managing this large qualitative data set truly transparent, underlying foundational components of

the study will be discussed. According to Sandelowski’s (2000) classifications, our study was a

qualitative descriptive study. The methodological underpinnings of this study were eclectic. The

qualitative component drew on tenets (e.g., importance of the setting and context, purposive

sampling, and inductive analysis) situated within the naturalistic paradigm espoused by Lincoln

and Guba (1985) and Miles and Huberman (1994). A subcomponent of the study (quantitative

data from surveys and regional corporate databases) was positioned within a positivist paradigm.

This latter component will not be addressed in this article.

Research questions focused on nurses’ and other healthcare providers’ perceptions of nurses

working to their full scope of practice. Participants were also asked to identify personal,

professional, and organizational barriers or facilitators that enabled or hindered their ability to

work to their full scope of practice. These types of questions are typically associated with

qualitative descriptive studies (Sandelowski, 2000). Data were collected on 14 acute care nursing

units located within three western Canadian Health Regions. To ensure variability in sampling,

patient care units from hospitals of various intensities and representing variability in provider and

organizational characteristics were selected across the health regions to participate in the study.

Types of units included in the study were intensive care, medicine, surgery, and psychiatry.

Individual, face-to-face, semi-structured interviews were conducted to gather information on

enactment of nursing roles and perceived facilitators and barriers to maximizing scope of

practice. A purposive volunteer sample of nursing personnel (e.g., RNs, LPNs, RPNs, and Patient

Care Managers) and inter-professional healthcare team members were recruited on the study

units. Patient interviews were also conducted with a small sample of volunteer patients from each

health region to validate the extent to which patient experience reflected the expected focus of

nursing defined in scope of practice documents. A total of 236 interviews were audio-recorded

and transcribed. There were 167 interviews conducted with nursing staff: 85 RNs, 31 LPNs, 11

RPNs, 19 patient care managers and assistant patient care managers, and 21 nurses in specialized

roles (e.g., nurse educators and nurse clinicians). The remainder of the interviews were completed

with other healthcare providers (e.g., physicians, social workers, and physiotherapists) and

patients. The establishment of rigour in this study became a daunting task when a large number of

nurses and other healthcare providers were interviewed to discuss enactment of scope of practice

International Journal of Qualitative Methods 2012, 11(3)

249

of nurses and the influence of the work environment and other structures and processes on role

enactment.

Ensuring Trustworthiness of the Data

Our study presented a myriad of challenges to ensure trustworthiness of the data. These

challenges included collecting data from multiple sites using different research assistants,

developing a process for analyzing the data, recruiting and retaining qualified individuals to

complete coding and initial analysis, and completing in-depth analysis of the data. Despite the

many challenges encountered while managing this large qualitative data set, we succeeded in

reporting research results that met the criteria for data trustworthiness. The following sections of

this article will outline the challenges and how they were addressed in handling the large amount

of data collected in this study.

Credibility or Authenticity

First, participant recruitment was an important aspect to ensure credibility of research results.

Participant selection required the incorporation of multiple perspectives (e.g., RNs, LPNs,

managers, and interdisciplinary team members) to provide a clear and broad understanding of

nurses’ and other healthcare providers’ perceptions of nursing scope of practice. Although

participants volunteered to be interviewed, researchers facilitated a diverse range of perspectives

by presenting in both posters and unit presentations the importance of capturing varied perception

of scope of practice. Maximum variation (Polit & Hungler, 1999) was sought when recruiting

units for the study. Variability in organizational characteristics, hospital intensity, and a variety of

patient care units was desired (Aita & McIlvain, 1999; Morse & Richards, 2002). Once hospital

units were selected, participants were recruited for interviews with attention to maximum

variability in representing the various perspectives of nursing scope of practice. Given the

geographical nature of the study, as well as the nature of the clinical environment, it was not

possible to limit interviews to one unit with one group of providers at a single point in time.

Rather, interviews were completed on various patient care units with a variety of providers in the

same time frame, depending on their work schedules. Initially, both research assistants and

several research team members were concerned that this process would result in more interviews

being conducted because of the uncertainty about data saturation. However, bi-weekly

discussions with interviewers confirmed that while they were hearing similar elements, data

saturation had not been reached. The variability in units, desired variability in participants, and

lack of data saturation until later in the study all led to the recruitment of a very large sample of

participants for the study, which created a situation requiring the management of a large

qualitative data set.

Second, the consistency of data collection was important to ensure credibility of research results.

Data collection was conducted across three health regions with multiple research assistants. Semi-

structured interview questions focused on components of the overarching research questions and

guided the interview process with nurses and interdisciplinary team members (Sandelowski,

2000). These interviews were conducted by three different research assistants. To standardize

interview tracking and scheduling and the entry of demographic data, an Access TM

database,

along with a user manual detailing the process and technical applications, were provided to each

study site. To further establish consistency in data collection (Aita & McIlvain, 1999; Morse &

Richards, 2002), a two day training session was conducted by the project manager with the

research assistants to discuss the interview protocol, review interview questions and cues, address

the concept of data saturation, and discuss the purpose of writing brief field notes following

interviews. A second component of the training session was the completion and review of several

International Journal of Qualitative Methods 2012, 11(3)

250

interviews with each of the research assistants. Feedback about interviewing techniques (e.g.,

paraphrasing, clarity, utilization of cues for questions, and getting the interviewee to elaborate on

their responses) was provided to each of the research assistants. At this time, research assistants

also shared their experiences in completing the interview and provided valuable feedback

contributing to the clarity of the interview questions and additional prompts for the questions. A

training manual was also provided as a resource for the interviewers to complement training

sessions. Given the magnitude of the study, research assistants were reminded to limit the number

of interviews (3-4) completed in one day to avoid interviewer burden. Despite the focus on

consistency of data collection, opportunity was provided through the field notes to co-author

results as noted in Kvale (1996).

Finally, presentations were made to participants to ensure the credibility and authenticity of the

research results. Ten to fifteen presentations were made to various groups of participants (e.g.,

nurses, allied healthcare professionals, Patient Care Managers, senior health leaders, and Chief

Nursing Officers) wherein results were validated by participants.

Dependability and Transferability

Data Processing and Cleaning

Consistency between transcribers was ensured by utilizing a consistent template, which permitted

easy transfer of documents into N6 TM

, a computer program designed for qualitative data storage,

indexing, and theorizing. Transcription guidelines to standardize expressions and formatting were

provided to each transcriptionist. To ensure the accuracy of the transcription, each of the research

assistants reviewed the transcripts while comparing them to audio files. Minor discrepancies, such

as spelling errors and clarification of acronyms, were made in the transcripts following review by

the research assistants.

Data Analysis

Preparation for the data analysis component of the study was intense, time consuming, and multi-

focused. A phased approach (Gaskell & Bauer, 2000) was used in data analysis, with the

completion of coding and initial analysis prior to in-depth analysis of the data. A conceptual

framework was developed by the research team to begin a content analysis process. As an initial

step in the development of the framework, two research team members and one research assistant

independently reviewed four interviews to identify preliminary themes. The second step in the

development of the framework was to have two of the original research team members and two

new research team members independently analyze four new interviews utilizing the existing

framework. With the analysis of this second set of interviews, consistencies were found in the

themes identified between the team members. However, new themes also emerged resulting in an

expansion of the conceptual framework. Miller and Crabtree (1992) have described this approach

to content analysis as a “template” style (p.18). The conceptual framework served as the initial

tree node structure to begin coding interviews in the N6 TM

software program.

As consistent with qualitative methods, the categories and nodes identified were not considered

static. Several iterations of this conceptual framework and tree node structure evolved,

particularly in the early stages of coding and analysis by the coders and the research team

(Gaskell & Bauer, 2000). In developing the evolving tree, particular attention was paid to the

semantic relationships of the parent and child nodes. A reference document defining each node

and indicating placement in the hierarchy of the tree structure was developed and modified to

reflect coding team discussions. This process assisted the dependability of the analysis of the

International Journal of Qualitative Methods 2012, 11(3)

251

large data set, which occurred across multiple coders. Documentation of the changes and the

rationale for changes were maintained to establish an audit trail (Lincoln & Guba, 1985).

Recruiting and retaining qualified individuals to do coding and initial analysis was both an

important and difficult task (Richards, 2005). Coders were recruited through the university and

connections with other qualitative researchers. Coding was accomplished through collaboration

and strengthened by the varying perspectives of multidisciplinary team members with

backgrounds in clinical and academic nursing, psychology, social work, occupational therapy,

and health services research. Most team members had previous qualitative research experience. A

half day of training was provided to all coders with ongoing consultation and assistance provided

by various members of the research team. Binders describing the technical aspects of N6 TM

were

developed for each coder. Initially the coders completed the coding of the same two interviews.

Coding was compared and the coding tree was discussed. Early in the coding process weekly

meetings were held with the coders; as the study progressed meetings were decreased to every

two weeks. These meetings provided an excellent opportunity both to discuss the development of

new themes and to question and confirm saturation of themes.

In-depth analysis was completed by two experienced qualitative research team members with

different healthcare backgrounds (nursing and occupational therapy). Analysis was completed by

provider group; each researcher examined the data for specific groups of providers (e.g., RNs,

LPNs, etc.). While each of the researchers examined descriptions of nursing scope of practice and

barriers and facilitators to enactment of scope of practice, patterns across the data were also

examined (Richards, 2005). Assigning data sets to different researchers (Gaskell & Bauer, 2000)

was seen as an appropriate and manageable approach to in-depth data analysis of this large data

set. Data overload, fatigue, and the potential for the researcher to “get lost in the data” posed real

challenges for the analysis of the large amount of data collected for this study. Dedicating two

researchers, who met and discussed regularly, to the process of in-depth analysis assisted in

managing these challenges. Researchers frequently documented their processes and

interpretations as memos directly in the software program. Summary analysis documents were

also created for each of the data sets analyzed. Meeting regularly was also important in making

meaningful sense of all the data in this study.

Although N6 TM

was an excellent program for organizing the qualitative data, challenges were

encountered in merging projects between coders (initial analysis) and again between researchers

once the in-depth analysis was completed. QSR Merge TM

is designed to merge one project with

another project, but with the coders and researchers each developing separate N6 TM

projects (five

different projects in total) merging was not seamless. Duplication of transcripts and codes

required that one individual be associated with one transcript to prevent duplication in the final

N6 TM

for the complete analysis of all the data for the study. Although challenges were

encountered in merging the projects, we were successful in creating one complete N6 TM

project.

Confirmability

Confirmability (Guba, 1981; Johnson & Waterfield, 2004) of research results was ensured via

four key processes: the creation of an audit trail; an internal audit; an external audit; and the

writing of the final research report.

Audit Trail

A detailed, comprehensive accounting of all data collection and data analysis activities was

completed. Changes were documented as they were made, along with rationale for the change.

International Journal of Qualitative Methods 2012, 11(3)

252

Accurate and comprehensive records of the methods employed in data collection and analysis by

researchers in the study were recommended by qualitative research experts (Lincoln & Guba,

1985; Sandelowski, 2000). Such audit trails provided not only a solid methodological reference

for the reader, but also provided an opportunity for reflective reasoning (on the themes or

categories chosen, interpretations, etc.) for the researchers as the study progressed (Guba, 1981;

Johnson & Waterfield, 2004; Whittemore et al., 2001). For example, if methodology changed at

some point in the study, an audit trail would keep a record of when, why, and what changes were

implemented. Such audit trails became especially useful for managing large data sets and placing

data, methodology, and interpretation within the particular context in which they belonged.

Internal audit

Internal audits of coding and themes for the study were completed at three different intervals

(after 10, 25, and 45 interviews were coded) during the analysis of the study. The purposes of

these audits were to assess inter-rater reliability and to determine similarities and differences in

key themes identified by coders and auditors. Audits were conducted by three auditors, each

members of the research team. The audit included interviews of nurses (RNs, RPNs, and LPNs),

nurse managers, interdisciplinary team members, and patients. The sample of interviews was

based on a stratified selection by profession, education, unit, and health region to ensure

maximum variability of codes and themes. Transcripts for review were then randomly selected

from these stratified data sets. Internal audit results are outlined in Table 1.

International Journal of Qualitative Methods 2012, 11(3)

253

Table 1: Internal audit results

Coder Auditor Interview Inter-rater Reliability Common Themes

Audit 1 (following 10 interviews coded by each coder) a

002 001 2055 Not applicable  Themes from auditors were compared to a summary of

initial findings

 Similar themes were found by both (e.g., lack of time,

fragmentation of care, lack of

role clarity and role definition,

role overlap) with the

exception of two additional

themes, one in the summary

(language) and one by auditors

reviewing interviews (job

stressors)

002 002 4414 76% accuracy

002 003 6041 Not applicable

003 002 6046 Not applicable

003 003 2029 74% accuracy

003 001 4106 Not applicable

004 003 4302 Not applicable

004 002 2028 Not applicable

004 001 6062 41% accuracy

Audit 2 (following 25 interviews coded by each coder) b

003 003 2000  Greater inter-rater reliability between coders than auditors

 Auditor 001consistently out of range as noted in Audit 1

 Auditor discrepancy likely related to different style of

coding, language, and

interpretation

 Positive coder reliability likely due to amount of coding

completed, interaction amongst

coders, and consistent

attendance at coder meetings

 Themes were compared to themes identified in a second

summary report to the

Advisory Committee

 Similarities and consistency in themes (e.g., time, role

overlap, importance of

communication, role clarity,

workload) were noted between

the audited interviews and the

summary report

003 003 2083

003 003 2011

004 001 6073

004 001 6060

004 002 4201

002 002 6035

002 002 6040

002,

003,

004

001,

002,

003

4203

Audit 3 (following 45 interviews coded by each coder)

002 001 2076  As inter-rater reliability was completed in prior audits, it was

not completed at this time

 Themes were compared to the final research report

 Themes were very similar (e.g., role overlap, role clarity,

time, continuity of care,

communication, workload),

although themes were

presented more broadly in the

final report

003 001 2036

004 001 6069

002 002 6017

003 002 2037

004 002 4308

002 003 2022

003 003 6076

004 003 4213 a=inter-rater reliability conducted on three randomly selected interviews from audit; compared coding of

coders to coders and auditors to coders

b=inter-rater reliability conducted on one interview; compared coding among auditors and one coder

International Journal of Qualitative Methods 2012, 11(3)

254

Overall, the internal audit showed positive results in inter-rater reliability of coding and common

themes identified from data analysis. Although discrepancies were found between coding

completed by coders and auditors, of note was the consistency in coding amongst coders. For the

research team this reliability emphasized the importance of the regular meetings with coders to

discuss node definitions and clarify where data elements best fit in the coding structure. The lack

of consistency in coding between auditors and coders was not unexpected. Coding of qualitative

data will be largely interpretive in nature; therefore, researchers’ insight and language will be

highly individual (Morse & Richards, 2002). The important finding in the internal audit was the

consistency in the themes identified from the data, which reinforced for the research team that the

right course was being pursued and the team should continue data analysis in the manner in which

it was being conducted. The internal audit also facilitated the opportunity for researchers to

engage with the data.

External audit

An expert in qualitative data analysis completed an external audit of the data. The reviewer was

not associated with the study in any way. Audit questions were developed from the work of Flick

(2002) and Miles and Huberman (1994) and reflected an assessment of the procedures undertaken

in the process of conducting the study. Questions are outlined in Table 2.

Table 2: External audit questions

External audit questions

1. Were the findings grounded in the data?

2. Were the inferences logical?

3. Were the category structures appropriate?

4. Were the decisions and methodological shifts justified?

5. Did researcher bias exist?

6. What strategies were used to increase credibility?

Overall, the external audit review was very favourable. The reviewer noted that the sample was

selected in a manner such that the units selected and the perspectives of various categories of

nurses were obtained. The summary reports provided to the external reviewer by the project

manager served as a complementary component to the data managed in N6 TM

; the connections

between identified categories and the data were easily accessible in a systematic manner. Team

meetings demonstrated that the reports were discussed at length, resulting in decisions based on

the data and documentation of changes to the framework.

Furthermore, the review confirmed that inferences made in the data were logical. More

specifically, there was sufficient data for the thematic categorical structures of assessment,

accountability, responsibility, coordination of care, general tasks, patient safety, patient

education, role overlap and ambiguity, autonomy, working to full scope, facilitators, barriers, and

recommendations for unit-based change. Conclusions drawn for these codes were very robust.

The reviewer did note that data were less developed for the codes of critical thinking, problem

solving, isolation, discontent, conflict, respect, and burnout.

The research team was commended for linking all inquiry decisions to the purpose and the

strategies of the study. Specific activities, such as attention to the multidisciplinary nature of the

International Journal of Qualitative Methods 2012, 11(3)

255

research team; bi-weekly meetings with coders; creation of a detailed audit trail; documentation

of the coding framework; and execution of an internal audit, were highlighted by the external

auditor as important in increasing the trustworthiness of the study. In terms of research bias,

while the researchers were commended for excellent use of follow-up questions to collect

additional descriptive information, it was suggested that deliberate recruitment of participants

who might hold contrary views to the researchers would have strengthened this component of the

review. Overall, the research team was commended for the data collection, analysis, and

interpretation of this very large qualitative study.

Research report

The final research report was written in such a way as to increase the confirmability of research

results. The report highlighted the shortcomings of the study and provided transparent links

between study results and the actual experiences of the participants in the study (Guba, 1981). To

this end, limitations of the study were outlined and quotations from participants were included to

represent themes identified in the study.

Strengths and Limitations

Several strengths of this study were noteworthy. First, given the large number of interviews

completed, a robust description of the scope of practice of nurses in acute care and the barriers

and facilitators impacting their ability to practice to full scope was clearly evident. During the

management and analysis of the data, we, as researchers, were reflexive and engaged in many

strategies that assisted us in questioning how our knowledge, position, and experience potentially

influenced or shaped analysis and interpretation of research results (Pyett, 2003). When the

findings of this study were discussed both formally and informally with nurses and other

professionals from jurisdictions across Canada, we found that the results seemed to resonate with

those colleagues. We, therefore, are reasonably confident that the findings from this research

represented a current state that potentially characterizes many health care settings.

Several limitations in the methodology to manage the qualitative data for this study were

identified. One key limitation was the inability to simultaneously analyze the data in an iterative

manner to inform the interview process. This was difficult because data were collected by three

research assistants across three geographically diverse sites. Working across sites was particularly

challenging. Timelines were also difficult to manage given the magnitude of the study.

Conclusion

Although there were a variety of challenges in managing the large volume of data generated by

the large number of interviews, the external audit report confirmed for the research team the

strengths of the strategies implemented to manage the data and ensure the quality of the data

analysis. We believe that the collective attention to data collection and analysis—via the training

of interviewers and coders, careful development of the coding framework, expertise of the

qualitative researchers in the analysis of the data, and attention to the development of an audit

trail— has contributed to a rich description of the scope of practice of nursing providers and the

barriers and facilitators to enactment of their scope of practice. Both the internal and external

audit also demonstrated the researchers’ commitment to remaining true to the findings. As

emphasized in the external audit, researchers utilized rigorous methodology both to manage the

data and to ensure that the data analysis captured the unique experience of participants (Ayres,

Kavanaugh, & Knafl, 2003). These data management methodologies have been employed as a

International Journal of Qualitative Methods 2012, 11(3)

256

template for other large research studies in which data were collected across sites with multiple

interviewers, participants, and coders.

The research team makes eight recommendations to help ensure rigour in the management of

large scale qualitative studies. First, the importance of the organization of the study cannot be

underestimated. One person must take on the role of managing the study. The organization of

staff, scheduling, data collection, data analysis, and the data itself is essential to the success of the

project. Second, diligent documentation of data collection and analysis details (e.g., changes in

approach and rationale) is required. This responsibility is best assumed by one person on the

research team. Third, ensuring a strict timeline for data collection, coding, and analysis is

essential. Fourth, make every effort to use an iterative process for data collection and analysis.

Fifth, conduct, at a minimum, a comprehensive internal audit at key points throughout the study.

We would encourage researchers to undertake an external audit to further increase the credibility

of the study. An external audit also provides an excellent learning opportunity for research team

members. Sixth, regular communication between team members is critical to ensure quality

completion of the study. Regular email contact, phone conversations, and face-to-face and

teleconference meetings are recommended. Seventh, adequate resources are required to ensure

timeliness and quality of results. Resources include both financial and human resources. Finally,

maintain a good sense of humour and build in time to reflect and have fun. The commitment to

large qualitative research is enormous and requires a team effort, with diversion from time to

time.

There is a lack of scientific literature regarding the structures and processes for managing large

qualitative data sets. This article provides concrete examples and recommendations for managing

these large scale qualitative studies to ensure rigour of study results. The external audit completed

by an expert qualitative researcher validates the processes and confirms the successful

management of this large data set and research study. This information will be invaluable as

researchers continue to answer complex health services research questions that inevitably result in

large qualitative data sets.

International Journal of Qualitative Methods 2012, 11(3)

257

References

Advisory Committee on Health Human Resources. (2002). Our health, our future: Creating

quality workplaces for Canadian nurses. Final report of the Canadian Nursing Advisory

Committee. Ottawa, ON: Health Canada.

Aita, V. A., & McIlvain, H. E. (1999). An armchair adventure in case study research. In B. F.

Crabtree & W. L. Miller (Eds.), Doing qualitative research (2 nd

ed., pp. 253-268).

Thousand Oaks, CA: Sage.

Ayres, L., Kavanaugh, K., & Knafl, K. A. (2003). Within-case and across-case approaches to

qualitative data analysis. Qualitative Health Research, 13, 871-883.

Creswall, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods

approaches (3 rd

ed.). Thousand Oaks, CA: Sage.

Flick, U. (2002). Introduction to qualitative research (2 nd

ed.). Thousand Oaks, CA: Sage.

Fyke, K. (2001). Caring for Medicare: Sustaining a quality system. Saskatchewan Commission

on Medicare. Regina, SK: Government of Saskatchewan.

Gaskell, G., & Bauer, M. (2000). Towards public accountability: Beyond sampling, reliability

and validity. In M. Bauer & G. Gaskell (Eds.), Qualitative researching with text, image

and sound (pp. 336-350). London, UK: Sage.

Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries.

Educational Communication and Technology Journal, 29, 75-91.

Johnson, R., & Waterfield, J. (2004). Making words count: The value of qualitative research.

Philosophy Research International, 9, 121-131.

Knafl, K. A., & Ayres, L. (1996). Managing large qualitative data sets in family research. Journal

of Family Nursing, 2, 350-364.

Kvale, S. (1996). An introduction to qualitative research interviewing. Thousand Oaks, CA: Sage.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage.

Miles, M., & Huberman, A. (1994). Qualitative data analysis: An expanded sourcebook (2 nd

ed.).

Thousand Oaks, CA: Sage.

Miller, W. L., & Crabtree, B. F. (1992). Primary care research: A multimethod typology and

qualitative road map. In B. F. Crabtree & W. L. Miller (Eds.), Doing qualitative research

(pp. 3-28). Newbury Park, CA: Sage.

Morse, J. M., & Richards, L. (2002). Readme first for a user’s guide to qualitative methods.

Thousand Oaks, CA: Sage.

Oelke, N. D., White, D., Besner, J., Doran, D., McGillis-Hall, L., & Giovannetti, P. (2008).

Nursing workforce utilization: An examination of facilitators and barriers on scope of

practice. Nursing Leadership, 10(1), 58-71.

International Journal of Qualitative Methods 2012, 11(3)

258

Polit, D. F., & Hungler, B. P. (1999). Nursing research: Principles and methods (6 th ed.).

Philadelphia, PA: Lippincott Williams & Wilkins.

Pyett, P. (2003). Validation of qualitative research in the “real world.” Qualitative Health

Research, 13, 1170–1179.

Richards, L. (2005). Handling qualitative data: A practical guide. London, UK: Sage.

Sandelowski, M. (1997). “To be of use”: Enhancing the utility of qualitative research. Nursing

Outlook, 45(3), 125-132.

Sandelowski, M. (2000). Whatever happened to qualitative description? Research in Nursing &

Health, 23, 334-340.

White, D., Oelke, N. D., Besner, J., Doran, D., McGillis-Hall, L., & Giovannetti, P. (2008).

Nursing scope of practice: Descriptions and challenges. Nursing Leadership, 10, 44-57.

Whittemore, R., Chase, S., & Mandle, C. (2001). Validity in qualitative research. Qualitative

Health Research, 11, 522-537.

Copyright of International Journal of Qualitative Methods is the property of International Institute for

Qualitative Methodology and its content may not be copied or emailed to multiple sites or posted to a listserv

without the copyright holder's express written permission. However, users may print, download, or email

articles for individual use.