U8A1-68 - TUTOR PLEASE FOLLOW THE INSTRUCTIONS AS OUTLINED TO COMPLETE THIS HOMEWORK....Data Analysis Strategies...SEE ATTACHED FILES
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 1/37
EXHIBIT 8.21 Immigration Roadmap Into the United States
Based on interviews, document analysis, case studies, and key informants’ expertise, this diagram depicts the barriers to foreigners getting a green card in the United States as of 2009. Describing (in words) what is on this one-page diagram would take 30 to 50 pages. The daunting nature of the visual and the seven “Sorry” hexagons that represent failure give a visceral feel to what it would be like to be caught somewhere in this maze.
SOURCE: ImmigrationRoad.com (2013). Used with permission.
EXHIBIT 8.22 Interpersonal Systems
In a program for low-income, drug-addicted, pregnant teenagers, each young woman was asked to draw a map of her closest relationships as they were at the start of the program (baseline)—people who could support her during her pregnancy and after her baby was born. The program aimed to strengthen that network of support in keeping with the mantra that it takes a village. The interpersonal maps show the diversity of relationships these young women had, including one who was completely isolated and alone (Map 3). The program used the maps to help the program participants map the strength of their interpersonal support systems. The maps were used in the evaluation to graphically depict change, accompanied by a case study of each young woman.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 2/37
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 3/37
SOURCE: McDonnell, Tran, and McCoy (2010).
EXHIBIT 8.23 Mountain of Accountability
I have worked with the The Blandin Foundation in Grand Rapids, Minnesota, on and off, over more than 30 years. In 2013, the senior staff undertook a strategic reflective practice exercise that included examining and making sense of the many different kinds of evaluation going on at the foundation. A lot was happening, but the pieces didn’t seem to fit together. As we examined the various elements and what each did, we looked for a metaphor or graphic depiction that would bring order and cohesion to the messiness and multifaceted evaluation functions. What emerged was a Mountain of Accountability, based on Maslow’s hierarchy framework, where basic necessities are at the bottom and organizational excellence is at the top. Information and patterns from lower levels inform higher levels, and understandings, lessons, and insights from higher levels flow back down to support lower levels and facilitate adaptation and ongoing learning.
(For the full explanation of the Mountain of Accountability, see the website http://blandinfoundation.org/who-we-are/accountability.php.)
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 4/37
EXHIBIT 8.24 Depicting Interconnected Factors
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 5/37
In the discussion on causal attribution, I featured the example of how the GEM was used to determine how an advocacy campaign might have contributed to and influenced a U.S. Supreme Court decision. The preponderance of evidence pointed to some influence (see p. 595). Based on our synthesis of the findings, we generated a systems model depicting the interdependent elements of an integrated approach to policy reform that makes coalition building a centerpiece strategy. The model consists of six factors that, together, contribute to strengthening policy reform. There are six elements in the generic model:
1. Strong, high-capacity coalitions: Working through coalitions is a common centerpiece of advocacy strategy.
2. Strong national–state grassroots coordination: Effective policy change coalitions in the United States have to be able to work bottom up and top down, with national campaigns supporting and coordinating state and grassroots efforts while state efforts infuse national campaigns with local knowledge and grassroots energy. Strengthening strong national–state coordination is part of coalition development and field building.
3. Disciplined and focused messages with effective communication: Effective communication must occur within movements (message discipline) and to target audiences (focused messaging). Strengthening communication has been a key component of advocacy coalition building.
4. Solid research and knowledge base: The content of effective messages must be based on solid research and timely knowledge. In the knowledge age, policy coalitions must be able to marry their values with relevant research and real-time data about the dynamic policy environment.
5. Timely, opportunistic lobbying and judicial engagement: The evaluation findings emphasize that effective lobbying requires connections, skill, flexibility, coordination, and strategy.
6. Collaborating funders engaged in strategic funding: Effective funding involves not only financial support but also infusion of expertise and strategy as part of field building.
Overall Lesson for Effective Advocacy
In essence, strong national–state–grassroots coordination depends on having a high-capacity coalition. A solid knowledge and research base contributes to a focused message and effective communication. Message discipline depends on a strong coalition and national–state coordination, as does timely and opportunistic lobbying and judicial engagement. To build and sustain a high-capacity coalition, funders must use their resources and knowledge to collaborate around shared strategies. These factors in combination and as mutual reinforcement strengthen advocacy efforts. In classic systems framing, the whole is greater than the sum of the parts, and the optimal functioning of each part depends on the optimal integration and integrated functioning of the whole.
We needed a graphic depicting the dynamic nature of the relationship among these factors, basically depicting the interconnections among the factors as looking like a fluid spider web.
The interdependent system of factors that contribute to effective advocacy and change
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 6/37
EXHIBIT 8.25 Depicting Organizational Tensions
An organizational development study revealed a number of fundamental tensions. The study was based on interviews, focus groups, and extensive review of documents, and observations of meetings and operational implementation and program delivery sessions. To report the findings, we created a set of graphic representations highlighting the tensions. Two ovals represent the two arenas of action that are in tension (e.g., strategy vs. execution). Within each oval are listed key elements that define each strategic dimension (i.e., key elements of strategy in one oval and key elements of execution in the other oval). Connecting the two ovals is a large arrow that shows the two dimensions as interrelated. Highlighted between the two ovals are some of the principal tensions we identified between them.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 7/37
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 8/37
EXHIBIT 8.26 Depicting Evaluation Tensions
No tension more deeply permeates evaluation than the admonition that evaluators should work closely with stakeholders to enhance mutual understanding, increase relevance, and facilitate use while maintaining independence to ensure credibility. Managing this tension was at the center of the way the evaluation of the Paris Declaration on Development Aid was structured, administered, and governed.
• At the country level, the national reference group in each country selected the evaluation team, based on published terms of reference and competitive processes; country evaluation teams were to operate independently in carrying out the evaluation.
• At the international level, the secretariat administered the evaluation, with an evaluation management group that selected the core evaluation team and provided oversight and guidance, while the core evaluation team conducted the evaluation independently.
• The international reference group of country representatives, donor members, and international organization participants provided a stakeholder forum for engaging with the core evaluation team and the evaluation’s draft findings.
Thus, at every level, the evaluation was structured to separate evaluation functions from political and management functions, while also providing forums and processes for meaningful interaction. Such structures and processes are fraught with tensions and risks. Yet, in the end, the credibility and integrity of the evaluation depended on doing both well: engaging key stakeholders to ensure relevance and buy-in while maintaining independence to ensure the credibility of findings. In our report, we created a graphic to depict this tension (Patton & Gornick, 2011a, p. 37).
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 9/37
The Paris Declaration evaluation team committed to a bottom-up design with country evaluations and donor studies feeding into a final synthesis of findings. But to make the synthesis manageable and facilitate aggregation of findings, the evaluation approach included building overarching, common elements:
• A common evaluation framework • Common core questions • A standardized data collection matrix • Standardized scales for rendering judgments • Standardized analytical categories for describing and interpreting progress: direction of travel,
distance traveled, and pace
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 10/37
• A common glossary of terms
So the evaluation had a top-down, standardized framework to facilitate synthesis. The joint, collaborative, and participatory nature of the evaluation meant that these bottom-up and top-down processes had to be carefully managed. We found some confusion and tension about the relationship between country evaluations and the overall global synthesis, so we created a graphic to summarize, highlight, and depict the tensions that emerged in the data (Patton & Gornick, 2011a, p. 51).
Strengths and Weaknesses of Visualization
The visualization examples I’ve shared are not meant to stand alone. They supplement and focus qualitative reporting but are not a substitute for narrative. Exhibit 8.27 concludes this section with a graphic overview of strengths and weaknesses in visualizing qualitative data and findings.
EXHIBIT 8.27 Visualization of Qualitative Data and Findings: Strengths and Weaknesses
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 11/37
Resources for Qualitative Data Visualization
• Azzam and Evergreen (2013a, 2013b). Data Visualization • Ball and Smith (1992). Analyzing Visual Data • Banks (2007). Using Visual Data in Qualitative Research • Bryson, Ackermann, and Eden (2014). Visual Strategy: A Workbook for Strategy Mapping for Public
and Nonprofit Organizations • Bryson, Ackerman, Eden, and Finn (2004). Visible Thinking • Davis (2014). Conversations About Qualitative Communication Research • Evergreen (2012). “Potent Presentations” • Heath et al. (2010). Video in Qualitative Research: Analyzing Social Interaction in Everyday Life • Henderson and Segal (2013). Visualizing Qualitative Data in Evaluation • Kistler (2014). “Innovative Reporting” • Margolis and Pauwels (2011). The SAGE Handbook of Visual Research Methods • Mathison (2009). “Seeing Is Believing: The Credibility of Image-Based Research and Evaluation” • Miles et al. (2014). Qualitative Data Analysis: A Methods Sourcebook
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 12/37
SOURCE: Brazilian cartoonist Claudius Ceccon. Used with permission.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 13/37
MODULE
74 Special Analysis and Reporting Issues: Mixed Methods,
Focused Communication, Principles-Focused Report Exemplar, and Creativity
For some critics, qualitative research has been getting better and more inventive. Even such a demanding veteran as Van Maanen suggests that ethnographers, especially those in the vanguard of the new ethnographic genres, are “learning to write better, less soothing, more faithful and ultimately more truthful accounts of their fellow human beings than ever before.” And Denzin sees methodological blossoming in the cross-fertilization of new “epistemologies . . . and new genres.”
—Huberman and Miles (2002, pp. ix–x) The Qualitative Researcher’s Companion
Integrating Qualitative and Quantitative Data
Mixed methods researchers are extending our understandings of how to understand complex social phenomena, as well as how to use research to develop effective interventions to address complex social problems.
—Donna M. Mertens (2013) Editor, Journal of Mixed Methods Research
“Emerging Advances in Mixed Methods: Addressing Social Justice”
Exhibit 5.8, on purposeful sampling (pp. 266–272), includes several mixed-methods strategies for picking cases. Here, again, design frames analysis. Thus, a mixed-methods design anticipates a mixed-methods analysis. For example, a random, representative sample of recent immigrants in Minnesota was surveyed to determine their priority needs. A small number of respondents in different categories of need were then interviewed to understand in depth their situations and priorities and to make the statistical findings more personal through stories. The qualitative sample was also used to validate the accuracy and meaningfulness of the survey responses.
Mixed methods have increased in both importance and credibility as the strengths and weaknesses of both quantitative and qualitative approaches have become more evident. Quantitative data are strong in measuring how widespread a phenomenon is. Qualitative methods are strong in explaining what the phenomenon means. Combining methods would seem straightforward. It is not. Often, mixed-methods reports are written in two sections, a qualitative section and a quantitative section, and the two self-contained studies never engage each other. The two reports, often written by two different people, are like toddlers engaging in parallel play in a sandbox. They’re old enough to recognize each other but not developed enough to interact. Ideally, however, studies are designed to be genuinely mixed (valuing both kinds of data) and, because the design makes it possible, the qualitative and quantitative data are integrated in analysis and reporting (Bergman, 2008; Greene, 2007; Mertens, 2013; Morgan, 2014). Exhibit 8.28 on pages 622–623 reviews challenges and solutions in mixed-methods analysis and reporting. But, first, to get into an appreciative mind-set for integrating qualitative and quantitative data, consider this intriguing example from the reflections of Abd Al- Rahman III, an emir and caliph of Córdoba in 10th-century Spain. He was an absolute ruler who lived in complete luxury. Here’s how he assessed his life:
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 14/37
I have now reigned above 50 years in victory or peace; beloved by my subjects, dreaded by my enemies, and respected by my allies. Riches and honors, power and pleasure, have waited on my call, nor does any earthly blessing appear to have been wanting to my felicity. I have diligently numbered the days of pure and genuine happiness which have fallen to my lot: They amount to 14. (Quoted by Brooks, 2014, p. SR1)
SIDEBAR
INSIGHTS FROM AN EXPERIENCED MIXED-METHODS EVALUATOR
Quantitative evidence is the bones; qualitative evidence is the flesh; and evaluative reasoning is the vital organs. If you are missing any of these you don’t have the full evaluative picture.
I frame all my evaluation work around asking and answering important evaluative questions. These are the “what are we trying to find out from this whole evaluation?” questions (not the interview questions, survey questions, etc, which are much lower level).
I’ve yet to conduct an evaluation that didn’t require some fairly substantial qualitative evidence to convincingly support the conclusions (answers to these high-level questions). In most cases I use mixed methods, but I have done a few evaluations that are 99–100% qualitative.
But more important than that, I use evaluation-specific methods. These are the ones that answer not just what’s happening and what’s changed, but the really tricky questions like whether the change is big enough, fast enough, worth the time, money and effort invested.
I also use qualitative methods a lot for causal inference. It’s a rare case where quantitative data alone makes a convincing case for a causal contribution, and qualitative evidence can be used to substantially strengthen the case—and in some cases is enough on its own.
—Jane Davidson Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation (2005)
Actionable Evaluation Basics: Getting Succinct Answers to the Most Important Questions (2012)
Cofounder of Genuine Evaluation blog (http://genuineevaluation.com/)
Focused Communications: Audience-Sensitive Presentations and Reports I emphasized earlier that reports should be written to communicate what you want to say to specific audiences. The particular challenge of qualitative reporting is reducing the sheer volume of data into digestible morsels. Book-length monographs may be the only way to do justice to the rich descriptions and detail that undergird long-term fieldwork and the in-depth case studies that in-depth interviews yield. But even if you get the opportunity to do a book, write a dissertation, or publish a full monograph, opportunities will arise for shorter and more focused articles and presentations. In such cases, focus is critical.
Even a comprehensive report will have to omit a great deal of information collected in a qualitative inquiry. If you try to include everything, you risk losing your readers or audience members in the sheer volume of the presentation. To enhance a report’s coherence or a presentation’s impact, follow the adage that less is more. This translates into covering a few key findings or conclusions well rather than lots of them poorly.
An evaluation report, for example, should address clearly each major evaluation question. The extent to which minor questions are addressed is a matter of negotiation with primary intended users. Sometimes,
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 15/37
answers to minor questions are provided orally or in an appendix to keep the focus on major questions. For each one, present the descriptive findings, analysis, and interpretation succinctly. An evaluation report should be readable, understandable, and relatively free of academic jargon. The data should impress the reader beyond the academic credentials of the evaluator.
©2002 Michael Quinn Patton and Michael Cochran
The advice I find myself repeating most often to students when they are writing articles or preparing presentations is FOCUS! FOCUS! FOCUS! The agony of omitting on the part of the qualitative researcher or evaluator is matched only by the readers’ or listeners’ agony in having to read or hear those things that were not omitted but should have been. Exhibit 8.29 presents an image of audience- and utilization-focused reporting.
EXHIBIT 8.28 Mixed-Methods Challenges and Solutions
The column on the left presents common challenges in mixing methods (qualitative and quantitative) during analysis and reporting. As summary items in tabular format, these are admittedly overstated and oversimplified contrasts. Those who are sophisticated about mixing methods will certainly find the contrasts exaggerated. That said, the literature on mixed methods (e.g., the Journal of Mixed Methods Research) and my own experience facilitating integration of different types of data lead me to emphasize the contrasts to highlight the potential and actual challenges of integration. Integrating different kinds of data is neither easy nor straightforward. That’s the core message of this exhibit. The column on the right, therefore, offers integrative strategies and solutions.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 16/37
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 17/37
Connecting With the Audience
An Evaluative Example Using Indigenous Typologies
Earlier in this chapter (pp. 546–548), I discussed analyzing findings through the framework of an indigenous typology—that is, using the language, concepts, and categories of the people studied to organize the data and make sense of the findings. Such an indigenous typology can also serve as an organizing framework for reports and presentations. Here’s an example.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 18/37
SIDEBAR
COLLABORATIVE OUTCOMES REPORTING
Collaborative Outcomes Reporting is a participatory approach to impact evaluation. Multiple kinds of evidence of outcomes are pulled together and presented as a “performance story.” To ensure the validity and credibility of the performance story, it is independently reviewed by both technical experts and knowledgeable stakeholders, which may include both program participants and community members. A collaborative approach to developing the performance story and review by independent experts are the distinctive elements of this approach (Dart & Andrews, 2014).
Collaborative Outcomes Report Structure
The report aims to explore and report the extent to which a program has contributed to outcomes.
In Collaborative Outcomes Reporting, reports are short and generally structured in terms of the following sections:
1. A narrative section explaining the program context and rationale 2. A “results chart” summarizing the achievements of the program against a program logic model 3. A narrative section describing the implications of the results, for example, the achievements (expected
and unexpected), the issues, and the recommendations 4. A section that provides a number of “vignettes” providing instances of significant change, usually
first-person narratives 5. An index providing more detail on the sources of evidence
COR [Collaborative Outcomes Reporting] is based on the premise that the values of stakeholders, program staff, and key stakeholders are of highest importance in an evaluation. The evaluators attempt to “bracket off” their opinions and instead present a series of data summaries to panel and summit participants for them to analyze and interpret. Values are surfaced and debated throughout the process. Participants debate the value and significance of data sources and come to agreement on the key findings of the evaluation. In addition, qualitative inquiry is used to capture unexpected outcomes, and deliberative processes are used to make sense of the findings. (Dart & Johnson, 2014)
EXHIBIT 8.29 Audience- and Utilization-Focused Reporting
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 19/37
Our evaluation of a rural leadership program included participant observation. The program involved a six- day residential retreat experience. After six days and evenings of intense (and sometimes tense) participation, observation, interviewing participants, and taking extensive field notes, our team of three evaluators needed a framework for providing feedback to program staff about what we were finding in a way that could be heard. We knew that the staff were heavily ego involved in the program and would be very sensitive to an approach that might appear to substitute our concept of the program for theirs. Yet a major purpose of the evaluation was to help them identify and make explicit their operating assumptions as evidenced in what actually happened during the six-day retreat.
As our team of three accumulated more and more data, debriefing each night what we were finding, we became increasingly worried about how to focus feedback. The problem was solved the fifth night when we realized that we could use their frameworks for describing to them what we were finding. For example, a major component of the program was having participants work with the Myers-Briggs Type Indicator, an instrument that measures individual personality type based on the work of Carl Jung (see Berens & Nardi, 1999; Hirsh & Kummerow, 1987; Kroeger & Thuesen, 1988; Myers & Meyers, 1995). The Myers-Briggs Type Indicator gives individuals scores on four bipolar scales:
In the feedback session, we began by asking the six staff members to characterize the overall retreat culture using the Myers-Briggs framework. The staff shared their separate ratings, on which there was no consensus, and then we shared our perceptions. We spent the whole morning discussing the database for and implications of each scale as a manifestation of the program’s culture. We ended the session by discussing where the staff wanted the program to be on each dimension. Staff were able to hear what we said without becoming defensive because we used their framework, a framework they defined, when giving individual feedback to participants, as nonjudgmental, facilitative, and developmental. Thus, they could experience our feedback as, likewise, nonjudgmental, facilitative, and developmental.
We formatted our presentation to staff using a distinction between “observations” and “perceived impacts” that program participants were taught as part of the leadership training. Observation: “You interrupted me in mid-sentence.” Perceived impact: “I felt cut-off and didn’t contribute after that.” This simple distinction, aimed at enhancing interpersonal communication, served as a comfortable, familiar format for program staff to receive formative evaluation feedback. Our reporting, then, followed this format. Three of the 20 observations from the report are reproduced in Exhibit 8.30.
EXHIBIT 8.30 Example of Reporting Feedback to Program Staff: Distinguishing Observations From Perceived Impacts Based on Their Indigenous Framework for Working With Participants in the Leadership Program
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 20/37
The critical point here is that we presented the findings using their categories and their frameworks. This greatly facilitated the feedback and enhanced the subsequent formative, developmental discussions. Capturing and using indigenous typologies can be a powerful analytical approach for making sense of and reporting qualitative data in evaluations.
The next example provides a quite different reporting format and illustrates a collaborative analysis and reporting effort.
Collaborative Reporting Example
A Principles-Focused Evaluation Report: An Example of the Flow of an Inquiry From Design to Data Collection and Analysis to Reporting
In 2012, six agencies serving homeless youth in Minneapolis and Saint Paul, Minnesota, began collaborating to evaluate the implementation and effectiveness of their shared principles. They first identified shared values and common principles and found that their work was undergirded and informed by eight principles. At the outset, these principles were just sensitizing concepts: labels and phrases without definition. Or where there was a descriptive definition, it varied among the agencies.
1. Trauma-informed care 2. Nonjudgmental engagement 3. Harm reduction 4. Trusting youth–adult relationships 5. Strengths-based approach
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 21/37
6. Positive youth development 7. Holistic approach 8. Collaboration
Working together, we designed a qualitative study to examine whether the principles actually guided the agencies’ work with youth and whether the outcomes the youth achieved could be linked to the principles. Nora Murphy, a graduate student at the University of Minnesota, was commissioned to conduct the study, which also served as her doctoral dissertation (Murphy, 2014). Selecting a maximum variation sample of homeless youth who had been involved with several of the agencies, Murphy generated 14 case studies of homeless youth based on interviews with the youth and program staff and review of case records. The results of the case studies were then synthesized. The collaborative participated in reviewing every case study to determine which principles were at work and their effectiveness in helping the youth meet their needs and achieve their goals. The story of Thmaris (not his real name) is one of the cases (see Exhibit 7.20, pp. 511– 516).
Coding the Case Studies and Interviews to Illuminate the Principles
The six agency directors participated in reviewing and validating the coding, as did the staff of the philanthropic foundation that funded the study. In pairs, they read through the case studies and identified passages, quotations, and events and experiences reported by the youth that seemed to illustrate a principle. Independently, as principal investigator, Murphy (2014) coded all the case studies against the sensitizing concept framework of the principles. Below, as an example, are three quotations from three different case studies (different youth) that were coded as illustrating the principle of trusting youth–adult relationships. (Pseudonym were selected by the youth.)
• Pearl: And you be like, “Okay, I have all this on my plate. I have to dig in and look into [the choices I’m making] to make my life more complete.” And I felt that on my own, I really couldn’t. Not even the strongest person on God’s green Earth can do it. I couldn’t do it. So I ended up reaching out to [the youth shelter], and they opened their arms. They were like just, “Come. Just get here,” and they got me back on track.
• Maria: If I was to sit in a room and think about, like, everything that happened to me or I’ve been through, I’ll get to cryin’ and feelin’ like I don’t wanna be on Earth anymore—like I wanted to die. When I talk to somebody [at the youth program] about it, it makes me feel better. The people I talk to about it give me good advice. They tell me how much they like me and how [good] I’m doin’. They just put good stuff in my head, and then I think about it and realize I am a good person and everything’s gonna work out better.
• Thmaris: [My case worker] is not going to send me to the next man, put me onto the next person’s caseload. He just always took care of me. . . . I honestly feel like if I didn’t have [him] in my corner, I would have been doing a whole bunch of dumb shit. I would have been right back at square one. I probably would have spent more time in jail than I did. I just felt like if it wasn’t for him, I probably wouldn’t be here right now, talking to you.
From Analysis to Reporting
Once all the case studies were coded against the principles, Murphy (2014) examined the data for additional principles. She found four possibilities and, in the end, confirmed one: being journey oriented in working with youth. In addition to the case studies, Murphy reviewed other available research (literature review for the dissertation) for definitions of and evidence of the effectiveness of each principle. Murphy used the cross-case analysis of the 14 interviews with the youth and the research literature to draft a definition for each principle. She and I then facilitated a process with the six agency directors to revise and finalize the definitions. In December 2013, the six agencies, acting together as a collaboration, formally adopted the nine principles (the original eight plus the one that emerged during the inquiry). Exhibit 8.31 presents the nine principles with the definitions developed and adopted. Next, the agency directors and Murphy (2014) worked together to
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 22/37
summarize the evidence, explaining each principle in a two-page statement. The outline for the report the group produced is presented in Exhibit 8.31. One of the most frequent questions I’m asked is how to consolidate, synthesize, and summarize all the rich data that come from a qualitative inquiry into a useful report that can be communicated publicly and used to inform practice and policy. There is no recipe for doing so, but Exhibit 8.31 provides an example to show that it can be done. Even though heavily edited, this is a lengthy exhibit. I have included it because of the importance of having some sense of how one moves from analysis to reporting when the purpose of the inquiry includes communicating results to a larger audience, as in evaluation and policy studies.
EXHIBIT 8.31 Principles-Focused Qualitative Evaluation Report Example
Nine Evidence-Based, Guiding Principles to Help Youth Overcome Homelessness
The Homeless Youth Collaborative on Developmental Evaluation
Part 1. The report begins with one page on each of the six agencies in the collaboration organized by type of service provided to homeless youth:
Outreach: StreetWorks
Drop-in Centers: Face to Face: SafeZone and YouthLink
Shelters: Avenues for Homeless Youth; Catholic Charities: Hope Street; and The Salvation Army: Booth Brown House
Other contributors to the process and the report
Funder: The Otto Bremer Foundation, Saint Paul, Minnesota
Technical assistance: Nora Muphy (2013) and Michael Quinn Patton
Part 2. The nine evidence-based, guiding principles to help youth overcome homelessness are presented.
The principles begin with the perspective that youth are on a journey; all of our interactions with youth are filtered through that journey perspective. This means we must be trauma-informed, non-judgmental and work to reduce harm. By holding these principles, we can build a trusting relationship that allows us to focus on youths’ strengths and opportunities for positive development. Through all of this, we approach youth as whole beings through a youth-focused collaborative system of support.
Journey Oriented: Interact with youth to help them understand the interconnectedness of past, present and future as they decide where they want to go and how to get there.
Trauma-Informed Care: Recognize that all homeless youth have experienced trauma; build relationships, responses and services on that knowledge.
Non-Judgmental Engagement: Interact with youth without labeling or judging them on the basis of background, experiences, choices or behaviors.
Harm Reduction: Contain the effects of risky behavior in the short-term and seek to reduce its effects in the long-term.
Trusting Youth-Adult Relationships: Build relationships by interacting with youth in an honest, dependable, authentic, caring and supportive way.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 23/37
Strengths-based Approach: Start with and build upon the skills, strengths and positive characteristics of each youth.
Positive Youth Development: Provide opportunities for youth to build a sense of competency, usefulness, belonging and power.
Holistic: Engage youth in a manner that recognizes that mental, physical, spiritual and social health are interconnected and interrelated.
Collaboration: Establish a principles-based, youth-focused system of support that integrates practices, procedures and services within and across agencies, systems and policies.
Part 3. Introduction to and explanation of principles-focused evaluation (excerpts)
All homeless young people have experienced serious adversity and trauma. The experience of homelessness is traumatic enough, but most also have faced poverty, abuse, neglect or rejection. They have been forced to grow up way too early. Most have serious physical or mental health issues. Some are barely teenagers; others may be in their late teens or early twenties.
Some homeless youth have family connections, some do not; all crave connection and value family. They come from the big city, small towns and rural areas. Most are youth of color and have been failed by systems with institutionalized racism and programs that best serve the white majority. Homeless youth are straight, gay, lesbian, bisexual, transgender or questioning. Some use alcohol or drugs heavily. Some have been in and out of homelessness. Others are new to the streets.
The main point here is that, while all homeless youth have faced trauma, they are all unique. Each homeless youth has unique needs, experiences, abilities and aspirations. Each is on a personal journey through homelessness and, hopefully, to a bright future.
Because of their uniqueness, how we approach and support each homeless young person also must be unique. No recipe exists for how to engage with and support homeless youth. As homeless youth workers and advocates, we cannot apply rigid rules or standard procedures. To do so would result in failure, at best, and reinforce trauma in the young person, at worst. Rules don’t work. We can’t dictate to any young person what is best. The young people know what is best for their future and need the opportunity to engage in self-determination.
This is where Principles come in. Organizations and individuals that successfully support homeless youth take a principles-based approach to their work, rather than a rules-based approach. Principles provide guidance and direction to those working with homeless youth. They provide a framework for how we approach and view the youth, engage and interact with them, build relationship with them and support them. The challenge for youth workers is to meet and connect with each young person where they are and build a supportive relationship from there. Principles provide the anchor for this relationship-building process.
Evaluation Design
Nora Murphy conducted 14 in-depth case studies of homeless youth who had interacted with the agencies, including interviews with the youth and program staff, and review of case records. The results of the case studies were then synthesized. The collaborative participated in reviewing every case study to determine which principles were at work and their effectiveness in helping the youth meet their needs and achieve their goals.
The results showed that all the participating organizations were adhering to the principles and that the principles were effective (even essential) in helping them make progress out of homelessness. While staff could not necessarily give a label to every principle at the time of the evaluation interviews, they clearly
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 24/37
talked about and followed them in practice. In the interviews with the youth, their stories showed how the staffs’ principles-based approaches made a critical difference in their journey through homelessness. (Reference for full dissertation: Murphy, 2013)
Part 4. Statements of 2–3 pages in which each principle is defined and discussed. For each principle, the summary statement explains why it matters, what key research shows, what the practice implications are, and concludes with illustrative quotes from the evaluation case studies.
Part 5. Following the principles statements are 2-page excerpts from each of the 14 case studies, excerpts that provide an overview of the outcomes the youth experienced through engagement with the youth- serving agencies.
Part 6. One full 15-page case study
Report Conclusion: Principles-Based Practice
Taken together, this set of principles provides a cohesive framework that guides practice in working with homeless youth.
For the full report, see Homeless Youth Collaborative on Developmental Evaluation (2014)
Nine Evidence-Based, Guiding Principles to Help Youth Overcome Homelessness (public report length, 71 pp.)
Use of the Report Findings and Case Studies
The six participating agencies are using the principles and case studies for program and staff development, recruitment and selection of new staff, policy advocacy on behalf of homeless youth, fund-raising, and expanded collaboration with other youth-serving organizations locally, statewide, and nationally. For further discussion of how and why qualitative inquiry is especially appropriate for evaluating principles-based programs and collaborations, see Chapters 4 (pp. 189–194) and 5 (p. 292)
Meeting the Challenge of Saying a Lot in a Small Space: The Executive Summary and Research Abstract
The executive summary is a fiction. —Robert Stake (1998, p. 370)
The fact that qualitative reports tend to be relatively lengthy can be a major problem when busy decision makers do not have the time (or, more likely, will not take the time) to read a lengthy report. Stake’s preference for insisting on telling the whole story notwithstanding is a preference I share, but my pragmatic, living-in-the-real-world side leads me to conclude that qualitative researchers and evaluators must develop the ability to produce an executive summary of one or two pages that presents the essential findings, conclusions, and reasons for confidence in the summary. The executive summary or research abstract is a dissemination document, a political instrument, and cannot be—nor is it meant to be—a full and fair representation of the study. An executive summary or abstract should be written in plain language, be highly focused, and state the core findings and conclusions. When writing the executive summary or research abstract, keep in mind that more people are likely to read that summary than any other document you produce.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 25/37
So here are a couple of tips:
1. Focus on findings not methods: I see lots of summaries and abstracts where it’s clear that the person reporting is more in love with his or her methods than his or her findings. But the reader is more likely to want to know about your findings than your methods. Report just enough about the methods to explain how the findings were generated, but highlight the most important findings.
2. Pilot test your drafts: Get some people you know to read your executive summary or abstract and tell you what it says to them.
3. Take time to write a well-crafted executive summary or abstract: French philosopher Blaise Pascal wrote a witty and oft-repeated apology in 1657: “I would have written a shorter letter if I had had more time.”
Carpe Diem Briefings As the hymn book is to the sound of music, the executive summary is to the oral briefing. Legendary are the stories of having spent a year of one’s life gathering data, poring over it, and writing a rigorous and conscientious report and then encountering some “decision maker” (I use the term lightly here) who says, “Well, now, I know that you put a lot of work into this. I’m anxious to hear all about what you’ve learned. I’ve got about 10 minutes before my next appointment.”
Should you turn heel and show him your backside? Not if you want your findings to make a difference. Use that 10 minutes well! Be prepared to make it count. Carpe Diem.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 26/37
MODULE
75 Chapter Summary and Conclusion, Plus Case Study Exhibits
The role of the qualitative researcher in research projects is often determined by the researcher’s stance and intent. . . . All senses are used to understand the context of the phenomenon under study, the people who are participants in the study and their beliefs and behaviors, and of course the researcher’s own orientation and purposes. . . . It is complicated, filled with surprises, and open to serendipity, and it often leads to something unanticipated in the original design of the research project. At the same time, the researcher works within the frame of a disciplined plan of inquiry, adheres to the high standards of qualitative inquiry, and looks for ways to complement and extend the description and explanation of the project through multiple methods of research, providing that this is done for a specific reason and makes sense. Qualitative researchers do not accept the misconception that more methods mean a better or richer analysis. Rather, the rationale for using selected methods is what counts. The qualitative researcher wants to tell a story in the best possible configuration.
—Valerie Janesick (2011, p. 176) “Stretching” Exercises for Qualitative Researchers
Chapter Overview and Summary This chapter opened by cautioning that no formula exists for transforming qualitative data into findings. Purpose drives analysis. Design frames analysis. But even with a clear purpose and a strong, appropriate design, the challenge of qualitative analysis remains: making sense of massive amounts of data. This involves reducing the volume of raw information, sifting trivia from significant data, identifying significant patterns, distinguishing signal from noise, and constructing a framework for communicating the essence of what the data reveal.
Exhibit 8.1 (pp. 522–523) offered 12 tips for laying a strong foundation for qualitative analysis. Exhibit 8.2 (p. 528) provided guidance in connecting design and analysis by linking purposeful sampling and purpose- driven analysis. Part of that connection is that in qualitative inquiry, analysis begins during fieldwork. Quantitative research texts, focused on surveys, standardized tests, and experimental designs, typically make a hard-and-fast distinction between data collection and analysis. But the fluid and emergent nature of naturalistic inquiry makes the distinction between data gathering and analysis far less absolute. In the course of fieldwork, ideas about directions for analysis will occur. Patterns take shape. Signals start to emerge from the noise. Possible themes spring to mind. Thinking about implications and explanations deepens the final stage of fieldwork. While earlier stages of fieldwork tend to be generative and emergent, following wherever the data lead, later stages bring closure by moving toward confirmatory data collection—deepening insights into and confirming (or disconfirming) patterns that seem to have appeared. Indeed, sampling confirming and disconfirming cases requires a sense of what there is to be confirmed or disconfirmed. Thus, ideas for making sense of the data that emerge while still in the field constitute the beginning of analysis.
Once formal analysis begins, the first task is organizing the raw data for analysis; then coding and content analysis begin systematically and rigorously. Qualitative software can help manage the data, but sense making remains a quintessential human responsibility. Analysis includes identifying and reflecting on patterns and themes that emerge during analysis as well as being reflective and reflexive about the analysis process. How do you know what you know? How do you make sense of the perspectives of the people you’ve observed and
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 27/37
interviewed? What do you know about the primary audience to whom your findings are addressed? These reflective questions are the doorway into reflexivity.
Constructing, analyzing, comparing, and interpreting case studies constitute core sense-making processes in many qualitative inquiries. Exhibit 8.33 (pp. 638–642), at the end of this chapter, presents an extensive example of an individual-level case study. Analytical approaches other than case studies are also part of qualitative analysis. Exhibit 8.10 (pp. 551–552) presented 10 types of qualitative analysis. Module 70 discussed interpreting findings, determining substantive significance, and elucidating phenomenological essence. Throughout the chapter, we’ve examined and discussed how and why analysis and reporting involve making distinctions between organizing data, presenting descriptive patterns and findings, interpreting themes, determining substantive significance, and, a particularly challenging challenge in qualitative inquiry, inferring causal relationships. Exhibit 8.19 (pp. 600–601) presents 12 approaches to qualitative causal analysis. The Halcolm graphic comic at the end of this chapter (pp. 635–637) offers an African fable to stimulate reflection on the nature of causal explanation.
Module 73 presented guidance on writing up and reporting findings, including incorporating creative visualizations that can powerfully and succinctly communicate core findings. Exhibits 8.20 to 8.27 presented an array of examples of visualization of findings. Exhibit 8.26 (pp. 616–618) summarized the strengths and weaknesses of visualization of qualitative data and findings. Exhibit 8.27 highlighted mixed-methods challenges and solutions, especially integrating qualitative and quantitative data and results. Exhibit 8.31 (pp. 627–628) presented an example of a qualitative report focused on principles for serving homeless youth. Exhibit 8.32 on the next page provides a checklist for data analysis, reporting, and interpretation that highlights some of the main points presented in this chapter. Exhibit 8.35 (pp. 643–649) presents excerpts from a qualitative report based on interviews.
Conclusion: The Creativity of Qualitative Inquiry
Creativity will dominate our time after the concepts of work and fun have been blurred by technology.
—Award-winning science fiction writer Isaac Asimov (1983, p. 42)
I have commented throughout this book that the human element in qualitative inquiry is both its strength and its weakness—its strength in allowing human insight and experience to blossom into new understandings and ways of seeing the world, its potential weakness in being so heavily dependent on the inquirer’s skills, training, intellect, discipline, and creativity. Because the researcher is the instrument of qualitative inquiry, the quality of the result depends heavily on the qualities of that human being. Nowhere does this ring more true than in analysis. Being an empathetic interviewer or astute observer does not necessarily make one an insightful analyst—or a creative one. Creativity seems to be one of those special human qualities that play an especially important part in qualitative analysis, interpretation, and reporting. Therefore, I close this chapter with some observations on creativity in qualitative inquiry.
I opened this chapter by commenting on qualitative inquiry as both science and art, especially qualitative analysis. The scientific part demands systematic and disciplined intellectual work, rigorous attention to details within a holistic context, and a critical perspective in questioning emergent patterns even while bringing evidence to bear in support of them. The artistic part invites exploration, metaphorical flourishes, risk taking, insightful sense making, and imaginative connection making. While both science and art involve critical analysis and creative expression, science emphasizes critical faculties more, especially in analysis, while art encourages creativity. The critical thinker assumes a stance of doubt and skepticism: Things have to be proven. Faulty logic, slippery linkages, tautological theories, and unsupported deductions are targets of the critical mind. The critical thinker studies details and looks beyond appearances to find out what is really
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 28/37
happening. Evaluators are trained to be rigorous and unyielding in critically thinking about and analyzing programs. Indeed, evaluation is built on the foundation of critical analysis.
Critical thinkers, however, tend not to be very creative. The creative mind generates new possibilities; the critical mind analyzes those possibilities, looking for inadequacies and imperfections. In summarizing research on critical and creative thinking, Barry F. Anderson (1980) has warned that the centrality of doubt in critical thinking can lead to a narrow, skeptical focus that hampers the creative ability to come up with innovative linkages or new insights.
EXHIBIT 8.32 Checklist for Qualitative Data Analysis, Interpreting Findings, and Reporting Results
1. Keep analysis grounded in the data. Return over and over to your field notes, interview transcripts, and the documents or artifacts you’ve collected that inform your analysis.
2. Present actual data for readers to experience firsthand. Use direct quotations, detailed observations, and document excerpts as the core of your reporting and as empirical support for your analysis and interpretations. “In this case, more is more. Convince your reader of your argument with evidence from transcripts, observations, reflective journals, and any other documentary evidence” (Janesick, 2011, p. 177).
3. Distinguish description from interpretation and explanation. Interpretation and explanation go beyond the data. Interpreting and explaining are part of the analyst’s responsibility, but the first and foremost responsibility is organizing and presenting accurate and meaningful descriptive findings.
4. Do justice to each case before doing cross-case analysis. This admonition follows from the previous one. Well-constructed, well-written, thorough, and rich case studies are worthy in and of themselves. Don’t let the allure of finding and presenting cross-case patterns and themes undercut full attention to the integrity and particularity of each case. And when distinguishing signal from noise, describe and interpret both the signal and the noise.
5. Purpose drives analysis and interpretation. If your purpose is to generate or test theory, then a significant part of your discussion will involve connecting your findings and analysis to inform and support theory-focused propositions. If your purpose is evaluative, to support program improvements and inform decision making about programs and policies, organize your findings to address core evaluation issues and provide actionable answers to evaluation questions.
6. Throughout the study track, document, and report your inquiry methods and analytical procedures. Findings flow from methods. Methods flow from inquiry questions. Inquiry questions flow from inquiry purpose. Connect these dots so that the linkages are clear. Report your methods and analytical process in sufficient detail that readers know from whence come your findings and interpretations and why you did what you did.
7. Be reflective and reflexive. As a qualitative inquirer, you are the instrument of the inquiry. Who you are, what brings you to this inquiry, your background, experience, knowledge, and training—all these matter. Report your reflexive inquiry in the methods section of your report (or elsewhere as appropriate). Acknowledge your prior beliefs. You should work to reduce your biases, but to say you have none is a sign that you have many. To state your beliefs up front—to say “Here’s where I’m coming from”—is a way to operate in good faith and to recognize that you perceive reality through a subjective filter. . . . Distinguishing the signal from noise requires both scientific knowledge and self-knowledge. (Silver, 2012, pp. 451, 453)
8. Work at the writing. How you present your findings will matter. Get editorial assistance as needed. Solicit feedback. Take the writing seriously. High school English teachers are right when they say that excellence in writing reflects excellence in thinking. All of your rigorous fieldwork and diligent analysis can come to naught if you present your findings poorly, incoherently, or in boring or error- plagued prose.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 29/37
9. Make the case for substantive significance. Substantive significance is a matter of judgment. It’s where you synthesize the evidence, consider its import, make transparent weaknesses and limitations in the data, balance strengths against weaknesses, weigh alternative possibilities, and state why the findings are important and merit attention. Don’t overstate the case. Don’t overgeneralize. But don’t become timid and afraid of naysayers. You know the data better than anyone. You’ve immersed yourself in them, worked with them, and lived with them. Say what you’ve come to know and what you don’t know, and provide the evidence for both judgments. Then, make the case for substantive significance. And, by all means, stay qualitative. Don’t determine significance by the numbers of people who said something. It’s not how many said something that matters. It’s the import, wisdom, relevance, insightfulness, and applicability of what was said, by many or by a few. (See MQP Rumination #8, pp. 557–560.)
10. Use both your critical and your creative faculties. Qualitative analysis offers myriad opportunities for both critical and creative thinking. These are not antagonists. Both capabilities dwell within you. Draw on both. Critical thinking is manifest in rigorous attention to alternative interpretations and explanations and in settling on those interpretations and explanations that best fit the preponderance of evidence. Creative thinking is manifest in well-told stories, evocative case studies, astute selection and juxtaposition of quotes and observations, and powerful, captivating, and accurate visual representations of the findings.
The critical attitude and the creative attitude seem to be poles apart. . . . On the one hand, there are those who are always telling you why ideas won’t work but who never seem able to come up with alternatives of their own; and, on the other hand, there are those who are constantly coming up with ideas but seem unable to tell good from the bad.
There are people in whom both attitudes are developed to a high degree . . . , but even these people say they assume only one of these attitudes at a time. When new ideas are needed, they put on their creative caps, and when ideas need to be evaluated, they put on their critical caps. (p. 66)
Qualitative inquiry draws on both critical and creative thinking—both the science and the art of analysis. But the technical, procedural, and scientific side of analysis is easier to present and teach. Creativity, while easy to prescribe, is harder to teach and perhaps harder to learn, but here’s some guidance derived from research and training on creative thinking (De Bono, 1999; Kelley & Littman, 2001; Patton, 1987, pp. 247– 248; Von Oech, 1998).
1. Be open: Creativity begins with openness to multiple possibilities. 2. Generate options: There’s always more than one way to think about or do something. 3. Diverge–converge–integrate: Begin by exploring a variety of directions and possibilities before focusing
on the details. Branch out, go on mental excursions, and brainstorm multiple perspectives before converging on the most promising.
4. Use multiple stimuli: Creativity training often includes exposure to many different avenues of expression: drawing, music, role-playing, story boarding, metaphors, improvisation, playing with toys, and constructing futuristic scenarios. Synthesizing through triangulation (see Chapter 9) promotes creative integration of multiple stimuli.
5. Side track, zigzag, and circumnavigate: Creativity is seldom a result of purely linear and logical induction or deduction. The creative person explores back and forth, round and about, in and out, over and under, and so on.
6. Change patterns: Habits, standard operating procedures, and patterned thinking pose barriers to creativity. Become aware of and change your patterned ways of thinking and behaving.
7. Make linkages: Many creative exercises include practice in learning how to connect the seemingly unconnected. The matrix approaches presented in this chapter push linkages. Explore linking qualitative
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 30/37
and quantitative data. Work at creative syntheses of case studies and findings. 8. Trust yourself: Self-doubt short-circuits creative impulses. If you say to yourself, “I’m not creative,” you
won’t be. Trust the process. 9. Work at it: Creativity is not all fun. It takes hard work, background research, and mental preparation. 10. Play at it: Creativity is not all work. It can and should be play and fun.
Drawing Conclusions and Closure
Certainty is false closure.
Ambiguity is honest closure.
Staying open eschews closure.
Enough is enough yields pragmatic closure.
Closure is hard. Not closing is harder.
Close . . . for now. —From Halcolm’s Benedictions
In his practical monograph Writing Up Qualitative Research, Wolcott (1990) considers the challenge of how to conclude a qualitative study. Purpose again rules in answering this question. Scholarly articles, dissertations, and evaluation reports have different norms for drawing conclusions. But Wolcott goes further by questioning the very idea of conclusions.
Give serious thought to dropping the idea that your final chapter must lead to a conclusion or that the account must build toward a dramatic climax. . . . In reporting qualitative work, I avoid the term conclusion. I do not want to work toward a grand flourish that might tempt me beyond the boundaries of the material I have been presenting or detract from the power (and exceed the limitations) of an individual case. (p. 55)
This admonition reminds us not to take anything for granted or fall into following some recipe for writing. Asking yourself, “When all is said and done, what conclusions do I draw from all this work?” can be a focusing question that forces you to get at the essence. Indeed, in most reports, you will be expected to include a final section on conclusions. But, as Wolcott (1990) suggests, it can be an unnecessary and inappropriate burden.
Or it can be a chance to look to the future. Spanish-born philosopher and poet George Santayana concluded thus when he retired from Harvard. Students and colleagues packed his classroom for his final appearance. He gave an inspiring lecture and was about to conclude when, in midsentence, he caught sight of a forsythia flower beginning to blossom in the melting snow outside the window. He stopped abruptly, picked up his coat, hat, and gloves, and headed for the door. He turned at the door and said emphatically, “Gentlemen, I should not be able to finish that sentence. I have just discovered that I have an appointment with April.”
Or as Halcolm would say, not concluding is its own conclusion.
I close this chapter with a practical reminder that both the science and the art of qualitative analysis are constrained by limited time. Some people thrive under intense time pressure, and their creativity blossoms. Others don’t. The way in which any particular analyst combines critical and creative thinking becomes partly a matter of style and partly a function of the situation, and it often depends on how much time can be found to play with creative possibilities. But exploring possibilities can also become an excuse for not finishing. There
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 31/37
comes a time for bringing closure to analysis (or a book chapter) and getting on with other things. Taking too much time to contemplate creative possibilities may involve certain risks, a point made by the following story (to which you can apply both your critical and your creative faculties).
The Past and the Future: Deciding in Which Direction to Look
A spirit appeared to a man walking along a narrow road. “You may know with certainty what has happened in the past, or you may know with certainty what will happen in the future, but you cannot know both. Which do you choose?”
The startled man sat down in the middle of the road to contemplate his choices. “If I know with certainty what will happen in the future,” he reasoned to himself, “then the future will soon enough become the past and I will also know with certainty what has happened in the past. On the other hand, it is said that the past is a prologue to the future, so if I know with certainty what has happened in the past, I will know much about what will happen in the future without losing the elements of surprise and spontaneity.” Deeply lost to the present in the reverie of his calculations about the past and future, he was unaware of the sound of a truck approaching at great speed. Just as he came out of his trance to tell the spirit that he had chosen to know with certainty the future, he looked up and saw the truck bearing down on him, unable to stop its present momentum.
—From Halcolm’s Evaluation Parables
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 32/37
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 33/37
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 34/37
EXHIBIT 8.33 Mike’s Career Education Experience: An Illustrative Case Study
Background
Sitting in a classroom at Metro City High School was difficult for Mike. In some classes, he was way behind. In math, he was always the first to finish a test. “I loved math and could always finish a test in about 10 minutes, but I wasn’t doing well in my other classes,” Mike explained.
He first heard about Experience-Based Career Education (EBCE) when he was a sophomore. “I really only went to the assembly to get out of one of the classes I didn’t like,” Mike confessed.
But after listening to the EBCE explanation, Mike was quickly sold on the idea. He not only liked the notion of learning on the job but also thought the program might allow him to work at his own speed. The notion of no grades and no teachers also appealed to him.
Mike took some descriptive materials home to his parents, and they joined him for an evening session at the EBCE learning center to find out more about the program. Now, after two years in the program, Mike is a senior, and his parents want his younger brother to get into the program.
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 35/37
Early EBCE testing sessions last year verified the inconsistency of Mike’s experiences in school. While his reading and language scores were well below the average scored by a randomly selected group of juniors at his school, he showed above-average abilities in study skills and demonstrated superior ability in math.
On a less tangible level, EBCE staff members early last school year described Mike as being hyperactive, submissive, lacking in self-confidence, and unconcerned about his health and physical appearance when he started the EBCE program. He was also judged to have severe writing deficiencies. Consequently, Mike’s EBCE learning manager devised a learning plan that would build his communication skills (in both writing and interpersonal relations) while encouraging him to explore several career possibilities. Mike’s job experiences and projects were designed to capitalize on his existing interests and broaden them.
First-Year EBCE Experiences. A typical day for Mike started at 8:00 a.m., just as in any other high school, but the hours in between varied considerably. On arriving at the EBCE learning center, Mike said he usually spent some time “fooling around” with the computer before he worked on projects under way at the center.
On his original application, Mike had indicated that his career preference would be computer operator. This led to an opportunity in the EBCE program to further explore that area and to learn more about the job. During April and May, Mike’s second learning-level experience took place in the computer department of City Bank Services. He broke up his time there each day into morning and afternoon blocks, often arriving before his employer-instructor did for the morning period. Mike usually spent that time going through computer workbooks. When his employer-instructor arrived, they went over flow charts together and worked on computer language.
Mike returned to the high school for lunch and a German class he selected as a project. EBCE students seldom take classes at the high school, but Mike had a special interest in German since his grandparents speak the language.
Following German class, Mike returned to the learning center for an hour of work on other learning activities and then went back to City Bank. “I often stayed there until 5:00 p.m.,” Mike said, even though high school hours ended at three.
Mike’s activities and interests widened after that first year in the EBCE program, but his goal of becoming a computer programmer was reinforced by the learning experience at City Bank. The start of a new hobby—collection of computer materials—also occurred during the time he spent at City Bank. “My employer-instructor gave me some books to read that actually started the collection,” Mike said.
Mike’s interest in animals also was enhanced by his EBCE experience. Mike has always liked animals, and his family has owned a horse since he was 12 years old. By picking blueberries, Mike was able to save enough to buy his own colt two years ago. One of Mike’s favorite projects during the year related to his horse. The project was designed to help Mike with basic skills and to improve his critical thinking skills. Mike read about breeds of horses and how to train them. He then joined a 4-H group with hopes of training his horse for a show.
Several months later, Mike again focused on animals for another EBCE project. This time he used the local zoo as a resource, interviewing the zoo manager and doing a thorough study of the Alaskan brown bear. Mike also joined an Explorer Scouting Club of volunteers to help at the zoo on a regular basis. “I really like working with the bears,” Mike reflected. “They were really playful. Did you know when they rub their hair against the bars it sounds like a violin?” Evaluation of the zoo project, one of the last Mike completed during the year, showed much improvement. The learning manager commented to Mike, “You are getting your projects done faster, and I think you are taking more time than you did at first to do a better job.”
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 36/37
Mike got off to a slow start in the area of life skills development. Like some of his peers, he went through a period described by one of the learning managers as “freedom shock,” when removed from the more rigid structure normally experienced in a typical school setting. Mike tended to avoid his responsibility to the more “academic” side of his learning program. At first, Mike seldom followed up on commitments and often did not let the staff know what he was doing. By the end of the year, he had improved remarkably in both of these behavior areas.
Through the weekly writing required in maintaining his journal, Mike demonstrated a significant improvement in written communications, both in terms of presenting ideas and feelings and in the mechanics of writing. Mike also noted an interesting change in his behavior. “I used to watch a lot of TV and never did any reading.” At the beginning of the following year, Mike said, “I read two books last year and have completed eight more this summer. Now I go to the book instead of the television.” Mike’s favorite reading material is science fiction.
Mike also observed a difference in his attitude to homework. “After going to school for six hours, I wouldn’t sit down and do homework. But in the EBCE program, I wasn’t sitting in a classroom, so I didn’t mind going home with some more work on my journal or projects.”
Mike’s personal development was also undergoing change. Much of this change was attributed to one of his employer-instructors, an elementary school teacher, who told him how important it is in the work world to wear clean clothes. Both she and the project staff gave Mike much positive reinforcement when his dressing improved. That same employer also told Mike that she was really interested in what he had to say and therefore wanted him to speak slower so he could be understood.
Mike’s school attendance improved while in the EBCE program. During the year, Mike missed only six days. This was better than the average absence for others in the program, which was found to be 12.3 days missed during the year, and much improved over his high school attendance.
Like a number of other EBCE students in his class, Mike went out on exploration-level experiences but completed relatively few other program requirements during the first three months of the school year. By April, however, he was simultaneously working on eight different projects and pursuing a learning experience at City Bank. By the time Mike completed his junior year, he had finished 9 of the required 13 competencies, explored nine business sites, completed two learning levels, and carried through on 11 projects. Two other projects were dropped during the year, and 1 is uncompleted but could be finished in the coming year.
On a more specific level, Mike’s competencies included transacting business on a credit basis, maintaining a checking account, designing a comprehensive insurance program, filing taxes, budgeting, developing physical fitness, learning to cope with emergency situations, studying public agencies, and operating an automobile.
Mike did not achieve the same level of success on all of his job sites. However, his performance consistently improved throughout the year. Mike criticized the exploration packages when he started them in the first months of the program and, although he couldn’t pinpoint how, said they could be better. His own reliance on the questions provided in the package was noted by the EBCE staff, with a comment that he rarely followed up on any cues provided by the person he interviewed. The packets reflected Mike’s disinterest in the exploration portion of EBCE work. They showed little effort and a certain sameness in the remarks about his impressions at the various sites.
Mike explored career possibilities at an automobile dealer, an audiovisual repair shop, a supermarket, an air control manufacturer, an elementary school, a housing development corporation, a city public works, a junior high school, and a bank services company.
Mike’s first learning-level experience was at the elementary school. At the end of three and one-half months, the two teachers serving as his employer-instructors indicated concern about attendance,
6/2/2018 Bookshelf Online: Qualitative Research & Evaluation Methods: Integrating Theory and Practice
https://online.vitalsource.com/#/books/9781483314815/cfi/6/42!/4/2/4/2@0:0 37/37
punctuality, initiative in learning, and the amount of supervision needed to see that Mike’s time was used constructively. Mike did show significant improvement in appropriate dress, personal grooming, and quality of work on assignments.
Reports from the second learning-level experience—at the computer department of the bank services company—showed a marked improvement. The employer-instructor there rated Mike as satisfactory in all aspects and, by the time of the final evaluation, gave excellent ratings in 10 categories—(1) attendance/punctuality, (2) adhering to time schedules, (3) understanding and accepting responsibility, (4) observing employer rules, (5) showing interest and enthusiasm, (6) showing poise and self-confidence, (7) using initiative in seeking opportunities to learn, (8) using employer site learning resources, (9) beginning assigned tasks promptly, and (10) completing the tasks assigned.