Research - 9 Discussion

Analyzing Qualitative Data

G3658-12

2003 Ellen Taylor-Powell

Marcus Renner

Program Development & Evaluation

Introduction Qualitative data consist of words and observa- tions, not numbers. As with all data, analysis and interpretation are required to bring order and understanding. This requires creativity, discipline and a systematic approach. There is no single or best way.

Your process will depend on:

■ the questions you want to answer,

■ the needs of those who will use the informa- tion, and

■ your resources.

This guide outlines a basic approach for analyz- ing and interpreting narrative data — often referred to as content analysis — that you can adapt to your own extension evaluations. For descriptions of other types of qualitative data analysis, see Ratcliff, 2002. Other techniques may be necessary for analyzing qualitative data from photographs and audio or video sources.

This booklet is a companion to Analyzing Quantitative Data G3658-6 in this series.

Narrative data Text or narrative data come in many forms and from a variety of sources. You might have brief responses to open-ended questions on a survey, the transcript from an interview or focus group, notes from a log or diary, field notes, or the text of a published report. Your data may come from many people, a few individuals, or a single case.

Any of the following may produce narrative data that require analysis.

■ Open-ended questions and written com- ments on questionnaires may generate single words, brief phrases, or full para- graphs of text.

■ Testimonials may give reactions to a program in a few words or lengthy com- ments, either in person or in written corre- spondence.

■ Individual interviews can produce data in the form of notes, a summary of the individ- ual’s interview, or word-for-word tran- scripts.

■ Discussion group or focus group inter- views often involve full transcripts and notes from a moderator or observer.

■ Logs, journals and diaries might provide structured entries or free-flowing text that you or others produce.

■ Observations might be recorded in your field notes or descriptive accounts as a result of watching and listening.

■ Documents, reports and news articles or any published written material may serve as evaluation data.

■ Stories may provide data from personal accounts of experiences and results of pro- grams in people’s own words.

■ Case studies typically include several of the above.

PD E&& University of Wisconsin-Extension

Cooperative Extension Madison, Wisconsin

The analysis process Once you have these data, what do you do? The steps below describe the basic elements of narra- tive data analysis and interpretation. This process is fluid, so moving back and forth between steps is likely.

Step 1 Get to know your data. Good analysis depends on understanding the data. For qualitative analysis, this means you read and re-read the text. If you have tape recordings, you listen to them several times. Write down any impressions you have as you go through the data. These impressions may be useful later.

Also, just because you have data does not mean those are quality data. Sometimes, information provided does not add meaning or value. Or it may have been collected in a biased way.

Before beginning any analysis, consider the quality of the data and proceed accordingly. Investing time and effort in analysis may give the impression of greater value than is merited. Explain the limitations and level of analysis you deem appropriate given your data.

Step 2 Focus the analysis. Review the purpose of the evaluation and what you want to find out. Identify a few key ques- tions that you want your analysis to answer. Write these down. These will help you decide how to begin. These questions may change as you work with the data, but will help you get started.

How you focus your analysis depends on the purpose of the evaluation and how you will use the results. Here are two common approaches.

Focus by question or topic, time period or event. In this approach, you focus the analysis to look at how all individuals or groups responded to each question or topic, or for a given time period or event. This is often done with open-ended ques- tions. You organize the data by question to look across all respondents and their answers in order to identify consistencies and differences. You put all the data from each question together.

You can apply the same approach to particular topics, or a time period or an event of interest. Later, you may explore the connections and rela- tionships between questions (topics, time periods, events).

Focus by case, individual or group. You may want an overall picture of:

■ One case such as one family or one agency.

■ One individual such as a first-time or teen participant in the program.

■ One group such as all first-time participants in the program, or all teens ages 13 to 18.

Rather than grouping these respondents’ answers by question or topic, you organize the data from or about the case, individual or group, and analyze it as a whole.

Or you may want to combine these approaches and analyze the data both by question and by case, individual or group.

Step 3 Categorize information. Some people refer to categorizing information as coding the data or indexing the data. However, categorizing does not involve assigning numeri- cal codes as you do in quantitative analysis where you label exclusive variables with preset codes or values.

To bring meaning to the words before you:

■ Identify themes or patterns — ideas, con- cepts, behaviors, interactions, incidents, terminology or phrases used.

■ Organize them into coherent categories that summarize and bring meaning to the text.

This can be fairly labor-intensive depending on the amount of data you have. But this is the crux of qualitative analysis. It involves reading and re-reading the text and identifying coherent categories.

You may want to assign abbreviated codes of a few letters, words or symbols and place them next to the themes and ideas you find. This will help organize the data into categories. Provide a descriptive label (name) for each category you create. Be clear about what you include in the category and what you exclude.

As you categorize the data, you might identify other themes that serve as subcategories. Continue to categorize until you have identified and labeled all relevant themes.

The following examples show categories that were identified to sort responses to the questions.

2 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N■ ■ ■

A N A L Y Z I N G Q U A L I T A T I V E D A T A ■ ■ ■

Question Categories Responses to the question were sorted into:

1. What makes a quality educational program? Staff (Stf), relevance (Rel), participation (Part), timeliness (Time), content (Con)

2. What is the benefit of a youth mentoring program? Benefits to youth (Y), benefits to mentor (M), benefits to family (Fam), benefits to community (Comm)

3. What do you need to continue your learning Practice (P), additional training (Trg), time (T), about evaluation? resources (R), feedback (Fdbk), mentor (M),

uncertain (U) Possible code abbreviations are designated in parentheses.

3

Here are two ways to categorize narrative data — using preset or emergent categories.

Preset categories You can start with a list of themes or categories in advance, and then search the data for these topics. For example, you might start with con- cepts that you really want to know about. Or you might start with topics from the research litera- ture.

These themes provide direction for what you look for in the data. You identify the themes before you categorize the data, and search the data for text that matches the themes.

Emergent categories Rather than using preconceived themes or cate- gories, you read through the text and find the themes or issues that recur in the data. These become your categories. They may be ideas or concepts that you had not thought about.

This approach allows the categories to emerge from the data. Categories are defined after you have worked with the data or as a result of working with the data.

Sometimes, you may combine these two approaches — starting with some preset cate- gories and adding others as they become apparent.

Your initial list of categories may change as you work with the data. This is an iterative process. You may have to adjust the definition of your cat- egories, or identify new categories to accommo- date data that do not fit the existing labels.

Main categories may be broken into subcategories. Then you will need to resort your data into these smaller, more defined categories. This allows for greater discrimination and differentiation.

For example, in the question about benefits of a youth mentoring program, data within the cate- gory benefits to youth might be broken into a number of subcategories.

Continue to build categories until no new themes or subcategories are identified. Add as many cat- egories as you need to reflect the nuances in the data and to interpret data clearly.

While you want to try to create mutually exclu- sive and exhaustive categories, sometimes sec- tions of data fit into two or more categories. So you may need to create a way to cross-index.

Reading and re-reading the text helps ensure that the data are correctly categorized.

Example 1 shows labeling of one open-ended question on an end-of-session questionnaire. In this example, all responses were numbered and given a label to capture the idea(s) in each comment. Later, you can sort and organize these data into their categories to identify patterns and bring meaning to the responses.

Question Categories

What is the benefit Benefits to youth (Y) of a youth mentoring School performance (Y-SP) program? Friendship (Y-Friends)

Self-concept (Y-SC) Role modeling (Y-RM)

Benefits to mentor (M) Benefits to family (Fam) Benefits tocommunity (Comm)

Subcategories

4 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N■ ■ ■

Example 1. Labeling data from an end-of-session questionnaire (21 respondents)

Categories: Practice (P), additional training (Trg), time (T), resources (R), feedback (Fdbk), mentor (M), uncertain (U)

Line 7 is left uncoded because

“Yes” is not usable data.

Step 4 Identify patterns and connections within and between categories. As you organize the data into categories — either by question or by case — you will begin to see patterns and connections both within and between the categories. Assessing the relative importance of different themes or highlighting subtle variations may be important to your analysis. Here are some ways to do this.

Within category description You may be interested in summarizing the infor- mation pertaining to one theme, or capturing the similarities or differences in people’s responses within a category. To do this, you need to assem- ble all the data pertaining to the particular theme (category).

What are the key ideas being expressed within the category? What are the similarities and differ- ences in the way people responded, including the subtle variations? It is helpful to write a summary for each category that describes these points.

Larger categories You may wish to create larger super categories that combine several categories. You can work up from more specific categories to larger ideas and concepts. Then you can see how the parts relate to the whole.

Relative importance To show which categories appear more impor- tant, you may wish to count the number of times a particular theme comes up, or the number of unique respondents who refer to certain themes. These counts provide a very rough estimate of relative importance. They are not suited to statis- tical analysis, but they can reveal general pat- terns in the data.

Relationships You also may discover that two or more themes occur together consistently in the data. Whenever you find one, you find the other. For example, youth with divorced parents consis- tently list friendship as the primary benefit of the mentoring program.

You may decide that some of these connections suggest a cause and effect relationship, or create a sequence through time. For example, respon- dents may link improved school performance to a good mentor relationship. From this, you might argue that good mentoring causes improved school performance.

Such connections are important to look for, because they can help explain why something occurs. But be careful about simple cause and effect interpretations. Seldom is human behavior or narrative data so simple.

Ask yourself: How do things relate? What data support this interpretation? What other factors may be contributing?

You may wish to develop a table or matrix to illustrate relationships across two or more cate- gories.

Look for examples of responses or events that run counter to the prevailing themes. What do these countervailing responses suggest? Are they important to the interpretation and understand- ing? Often, you learn a great deal from looking at and trying to understand items that do not fit into your categorization scheme.

Step 5 Interpretation – Bringing it all together Use your themes and connections to explain your findings. It is often easy to get side tracked by the details and the rich descriptions in the data. But what does it all mean? What is really important?

This is what we call interpreting the data — attaching meaning and significance to the analysis.

A good place to start is to develop a list of key points or important findings you discovered as a result of categorizing and sorting your data.

Stand back and think about what you have learned. What are the major lessons? What new things did you learn? What has application to other settings, programs, studies? What will those who use the results of the evaluation be most interested in knowing?

Too often, we list the findings without synthesiz- ing them and tapping their meaning.

Develop an outline for presenting your results to other people or for writing a final report. The length and format of your report will depend on your audience. It is often helpful to include quotes or descriptive examples to illustrate your points and bring the data to life. A visual display might help communicate the findings.

Sometimes a diagram with boxes and arrows can help show how all the pieces fit together. Creating such a model may reveal gaps in your investigation and connections that remain unclear. These may be areas where you can suggest further study.

5A N A L Y Z I N G Q U A L I T A T I V E D A T A ■ ■ ■

6 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N■ ■ ■

“Nuts and bolts” of narrative analysis Moving from a mass of words to a final report requires a method for organizing and keeping track of the text. This is largely a process of cutting and sorting.

Work by hand, either with a hard copy (print copy) or directly on the computer. Exactly how you manage the data depends on your personal preference and the amount and type of qualita- tive data you have. Here are some data manage- ment tips:

■ Check your data. Often, there are data from multiple respondents, multiple surveys or documents. Make sure you have everything together. Decide whether the data are of suf- ficient quality to analyze, and what level of investment is warranted.

■ Add ID numbers. Add an identification (ID) number to each questionnaire, respondent, group or site.

■ Prepare data for analysis. You may need to transcribe taped interviews. How complete to make your transcription depends on your purpose and resources. Sometimes, you may make a summary of what people say, and analyze that. Or certain parts of an interview may be particularly useful and important and just those sections are transcribed. Other times, you will want to have every word of the entire interview. However, transcription is time-consuming. So be sure both data quality and your use of the data are worth the investment.

With small amounts of narrative data, you may work directly from the original hard copy. However, text is usually typed into a computer program. In extension, we typically type into a word processing program (Microsoft Word or Word Perfect) or into Excel.

You may decide to use a relational data base management program such as ACCESS, or a special qualitative data analysis program.

Your decision depends on the size of your data set, resources available, preferences, and level of analysis needed or warranted.

Decide whether you will enter all responses ques- tion by question, or whether you want to keep all text concerning one case, individual, group or site together (see Step 2). Save the file.

If you type the data into a word processing program, it is helpful to leave a wide margin on the left so you have space to write labels for text and any notes you want to keep. Number each line to help with cutting and sorting later.

■ Make copies. Make a copy of all your data (hard copy and electronic files). This gives you one copy to work from and another for safekeeping.

■ Identify the source of all data. As you work with the data, you will need to keep track of the source of the information or the context of the quotes and remarks. Such information may be critical to the analysis. Make sure you have a way to identify the source of all the data, such as by individual, site and date.

Think about what information to keep with the data. For example, you might use identifiers to designate the respondent, group, site, county, date or other source information. Or you may wish to sort by variables such as age, gender or position. Will you want to compare and contrast by demographic variable, site and date?

These identifiers stay with the information as you cut and sort the data, either by hand or in the computer. If you are working with hard copies, you might use different colors of paper to color- code responses from different people or groups (for example, see Krueger, 1998).

■ Mark key themes. Read through the text. Look for key ideas. Use abbreviations or symbols (codes) to tag key themes — ideas, concepts, beliefs, incidents, terminology used, or behaviors. Or, you might give each theme a different color. Keep notes of emerg-

Computer software Several software programs — for example, Ethnograph and NUD*IST — specifically analyze qualitative data. They systematize and facilitate all the steps in qualitative analysis. SAS software will manipulate precategorized responses to summarize open-ended survey questions (see Santos, Mitchell and Pope, 1999). CDC EZ-Text is a freeware program developed by the Centers for Disease Control and Prevention.

For smaller data sets and modest analysis needs, many people work by hand, with a word processing program or spreadsheet.

Note: Mention of products is not intended to endorse them, nor to exclude others that may be similar. These are mentioned as a convenience to readers.

7A N A L Y Z I N G Q U A L I T A T I V E D A T A ■ ■ ■

ing ideas or patterns and how you are inter- preting the data. You can write or type these in the margins, or in a specified column. Or keep a separate notebook that records your thoughts and observations about the data (see Example 2).

■ Define categories. Organize or combine related themes into categories. Name (label) these categories by using your own descrip- tive phrases, or choose words and key phrases from the text. Be clear about what the category stands for. Would someone unfamiliar with the data understand the label you have chosen? Write a short description or definition for each category,

and give examples or quotes from the text that illustrate meaning. Check with others to see if your labels make sense. You may also describe what the category does not include to clarify what is included.

■ Cut and sort. Once you define categories and label data, grouping the data into cate- gories involves some form of cutting and sorting. This is a process of selecting sec- tions of data and putting them together in their category.

Hard copy — A simple method is to cut text out of the printed page and sort into differ- ent piles. Each pile represents a category and has a name. As you work with the data,

Example 2. Identify themes and label data.

Be responsive to local needs and questions

Availability Responsive: willing and able to answer questions, timeliness, personal touch

Local connection Follow-up

Geographic coverage Service area, serve same people, need to extend out Staff Serve community, professional, responsive

Focus set priorities; stretched too thin

Reaching out vs. focus

Staff = program

Create a wide margin where you can label key ideas.

Highlight quotes for future use.

Keep notes of emerging

ideas.

you may make new piles, combine piles, or divide piles into subcategories. Remember to keep the identifier (source of data) with the data so you know where the text came from. Also, remember that you are working with a copy, not the original material.

Electronic copy — It is relatively simple and fast to move text around in a word pro- cessing program using the Windows plat- form. You can cut and paste text into differ- ent Windows, each representing a single cat- egory. If you type the category label directly into the computer file, you can use the search function to gather chunks of text together to copy and paste. Or you can sepa- rate the text into paragraphs, code the beginning of each paragraph, and then sort the paragraphs. You may prefer to use Excel. If the data are in Microsoft Word, you can easily transfer them to Excel. Set up an Excel file that includes columns for the ID number, identifiers, categories (themes), codes, and text (see Example 3).

When cutting and sorting, keep track of the source of the data. Be sure to keep identifiers attached to all sections of data.

Keep enough text together so you can make sense of the words in their context. As you cut and move data, text can easily become frag- mented and lose its contextual meaning. Be sure to include enough surrounding text so the meaning is not open to misinterpretation.

If data do not seem to fit, place those in a sepa- rate file for possible use later.

■ Make connections. Once you sort the data, think about how the categories fit together and relate. What seems more important, less important? Are there exceptions or critical cases that do not seem to fit? Consider alter- native explanations. Explore paradoxes, con- flicting themes, and evidence that seems to challenge or contradict your interpretations.

To trace connections, you can spread note cards across a table, use sticky notes on walls, or draw diagrams on newsprint showing the categories and relationships. Another approach is to create a two-dimensional or three-dimensional matrix. List the categories along each axis, and fill the cells with corresponding evidence or data. For further explanation, see Patton, 1990.

You can use simple hand tabulations or a com- puter program:

■ to search and count the frequency a topic occurs or how often one theme occurs with another, or

■ to keep track of how many respondents touch on different themes.

Such counts may be illuminating and indicate relative importance. But treat them with caution — particularly when responses are not solicited the same way from all respondents, or not all respondents provide a response.

8 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N■ ■ ■

Example 3. Screen shot of Excel spreadsheet

Enhancing the process As with any analysis process, bias can influence your results. Consider the following ways to increase the credibility of your findings.

Use several sources of data. Using data from different sources can help you check your findings. For example, you might combine one-on-one interviews with information from focus groups and an analysis of written material on the topic. If the data from these dif- ferent sources point to the same conclusions, you will have more confidence in your results.

Track your choices. If others understand how you came to your con- clusions, your results will be more credible. Keep a journal or notebook of your decisions during the analysis process to help others follow your reasoning. Document your reasons for the focus you take, the category labels you create, revisions to categories you make, and any observations you note concerning the data as you work with the text.

People tend to see and read only what supports their interest or point of view. Everyone sees data through his or her own lens and filters. It is important to recognize and pay attention to this. The analysis process should be documented so that another person can see the decisions that you made, how you did the analysis, and how you arrived at the interpretations.

Involve others. Getting feedback and input from others can help with both analysis and interpretation. You can involve others in the entire analysis process, or in any one of the steps. For example, several people or one other person might review the data inde- pendently to identify themes and categories. Then you can compare categories and resolve any discrepancies in meaning.

You can also work with others in picking out important lessons once cutting and sorting is done. Or you can involve others in the entire analysis process, reviewing and discussing the data and their meaning, arriving at major conclu- sions, and presenting the results.

Involving others may take more time, but often results in a better analysis and greater ownership of the results.

Pitfalls to avoid Finally, with any qualitative analysis, keep in mind the following cautions.

Avoid generalizing. The goal of qualitative work is not to generalize across a population. Rather, a qualitative approach seeks to provide understanding from the respondent’s perspective. It tries to answer the questions: “What is unique about this indi- vidual, group, situation or issue? Why?”

Even when you include an open-ended question on a survey, you are seeking insight, differences, the individual’s own perspective and meaning. The focus is on the individual’s own or unique response.

Narrative data provide for clarification, under- standing and explanation — not for generalizing.

Choose quotes carefully. While using quotes can lend valuable support to data interpretation, often quotes are used that only directly support the argument or illustrate success. This can lead to using people’s words out of context or editing quotes to exemplify a point.

When putting together your final report, think about the purpose for including quotes. Do you want to show the differences in people’s com- ments, give examples of a typical response rela- tive to a certain topic, highlight success? In any event, specify why you chose the selected quotes. Include enough of the text to allow the reader to decide what the respondent is trying to convey.

Confidentiality and anonymity are also concerns when using quotes. Even if you do not give the person’s identity, others may be able to tell who made the remark. Consider what might be the consequences of including certain quotes. Are they important to the analysis and interpreta- tion? Do they provide a balanced viewpoint?

Get people’s permission to use their words. Check with others about the usefulness and value of the quotes you select to include.

Address limitations and alternatives. Every study has limitations. Presenting the prob- lems or limitations you had while collecting and analyzing the data helps others better under- stand how you arrived at your conclusions.

Similarly, it is important to address possible alternative explanations. What else might explain the results? Show how the evidence supports your interpretation.

9A N A L Y Z I N G Q U A L I T A T I V E D A T A ■ ■ ■

Concluding comments Working with qualitative data is a rich and enlightening experience. The more you practice, the easier and more rewarding it will become. As both a science and an art, it involves critical, analytical thinking and creative, innovative perspectives (Patton, 1990).

Be thoughtful, and enjoy.

References CDC EZ-Text. Centers for Disease Control and Prevention, National Center for HIV, STD, and TB Prevention Divisions of HIV/AIDS Prevention, Behavioral Intervention Research Branch. Retrieved 4-9- 03: http://www.cdc.gov/hiv/software/ez-text.htm

Krueger, Richard A. 1998. Analyzing and Reporting Focus Group Results. Focus Group Kit 6. Thousand Oaks, Calif.: Sage Publications.

Krueger, Richard A. 1988. Focus Groups: A Practical Guide for Applied Research. Newbury Park, Calif.: Sage Publications.

Miles, Matthew B., & A. Michael Huberman. 1994. Qualitative Data Analysis: An Expanded Sourcebook. Second Edition. Thousand Oaks, Calif.: Sage Publications.

Patton, Michael Q. 1990. Qualitative Evaluation and Research Methods. 2nd Edition. Newbury Park, Calif.: Sage Publications.

Pope, Catherine, Sue Ziebland & Nicholas Mays. 1999. Qualitative Research in Health Care. Second Edition. London: BMJ Publishing Group. Chapter 8. Analysing Qualitative Data. Retrieved 4-9-03: http://www.bmjpg.com/qrhc/chapter8.html

Ratcliff, Donald. 2002. Qualitative Research. Part Five: Data Analysis. Retrieved 4-9-03: http://www.don.rat- cliff.net/qual/expq5.html

Santos, J. Reynaldo A., Diann Mitchell & Paul Pope. 1999. Are Open-Ended Questions Tying You in Knots? Journal of Extension. 37:4. Retrieved 4-9-03: http://www.joe.org/joe/1999august/iw2.html

Resources This publication is one in a series of program evaluation guides designed to help extension educators better plan and implement credible and useful evaluations. These also may be useful to agencies or funders seeking realis- tic evaluation strategies.

These practical how-to evaluation publications are available on the UW-Extension Program Development and Evaluation web site:

www.uwex.edu/ces/pdande

This web site also houses Quick Tips, easy-to-use briefs for improving your evaluation practice. You can also find evaluation studies, instruments, workshop presen- tations, an evaluation curriculum and links to more resources. Maintained as part of the University of Wisconsin System, the web site is continually updated and improved.

10 P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N■ ■ ■

Note: Analyzing Qualitative Data is a companion to Analyzing Quantitative Data G3658-6 in this series.

© 2003 by the Board of Regents of the University of Wisconsin System. Send inquiries about copyright permissions to Cooperative Extension Publishing Operations, 103 Extension Bldg., 432 N. Lake St., Madison, WI 53706.

Authors: Ellen Taylor-Powell, evaluation specialist, and Marcus Renner, research assistant, Program Development and Evaluation, University of Wisconsin-Extension.

Acknowledgements: This booklet is based on material initially written in 1999 and reviewed by Dick Krueger (University of Minnesota), Rey Santos (Texas A&M University) and Heather Boyd (University of Wisconsin-Extension). Thanks go to them for their early input that hopefully is reflected in this final product that has been ably edited by Rhonda Lee.

Produced by Cooperative Extension Publishing Operations

University of Wisconsin-Extension, U.S. Department of Agriculture and Wisconsin counties cooperat- ing. UW-Extension provides equal opportunities in employment and programming, including Title IX and ADA. If you need this material in another format, contact the Office of Equal Opportunity and Diversity Programs or call Cooperative Extension Publishing Operations at (608) 262-2655.

Copies of this publication and others in this series are available from your Wisconsin county UW-Extension office or from Cooperative Extension Publications:

(877) 947-7827; Fax (414) 389-9130

http://www1.uwex.edu/ces/pubs

Analyzing Qualitative Data (G3658-12) I-04-2003

P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N■ ■ ■