Program Implementation & Evaluation

FROM: The Center for Civic Partnerships

SOURCE:

http://www.civicpartnerships.org/docs/tools_resources/Quan_Qual%20Methods%209.07.htm

Quantitative and Qualitative Evaluation Methods

Top Tips

Evaluation methods and the data they produce are grouped into two basic categories – quantitative and qualitative. In general, quantitative methods produce “hard numbers” while qualitative methods capture more descriptive data. The method(s) you choose are determined by the purpose(s) of your evaluation and the resources you have to design and conduct it. In practice, most researchers and evaluators agree that combining quantitative and qualitative techniques (sometimes called “mixed method” evaluations) produces a richer and more comprehensive understanding of a project’s accomplishments and learnings.

How are Quantitative and Qualitative Data Different? At the most basic level, data are considered quantitative if they are numbers and qualitative if they are words. Qualitative data may also include photos, videos, audio recordings and other non-text data. Those who favor quantitative data claim that their data are hard, rigorous, credible and scientific. Those in the qualitative camp counter that their data are sensitive, detailed, nuanced and contextual. Quantitative data best explain the why and how of your program, while qualitative data best explain the what, who and when.

Different techniques are used to collect and analyze quantitative and qualitative data:

Quantitative Techniques Qualitative Techniques Surveys/Questionnaires Observations Pre/post Tests Interviews Existing Databases Focus Groups Statistical Analysis Non-statistical (methods vary)

In general, evaluators agree that qualitative and quantitative data and methods have different strengths, weaknesses, and requirements that affect decisions about which methodologies are appropriate for which purposes. Quantitative Data and Evaluation Methods What are Quantitative Data?

 Pieces of information that can be counted mathematically

 Usually gathered by surveys from large numbers of respondents selected randomly

 Secondary data such as census data, government statistics, etc. often included in quantitative evaluations

 Analyzed using statistical methods

 Best used to answer what, when and who questions

 Not well suited to how and why questions Strengths Limitations

Findings can be generalized, if selection process well-designed and sample is representative of study population

Related secondary data sometimes not available, or accessing available data is difficult/impossible

Relatively easy to analyze Difficult to understand context of program activities

Data can be very consistent, precise, reliable

Data may not be robust enough to explain complex issues

Data collection is usually cost efficient

How do you collect Quantitative Data? Surveys

o Most common method o Self-administered or by someone else o Face-to-face, telephone, mail, web-based

Secondary Data

o Often used in conjunction with survey data o Includes census data, knowledge/attitude/behavior (KABB) studies, criminal justice statistics, performance

data, non-confidential client information, agency progress reports, etc.

Qualitative Data and Evaluation Methods What are Qualitative Data?

 Usually gathered by observations, interviews or focus groups

 May also be gathered from written documents and through case studies

 Less emphasis on counting numbers of people who think or behave in certain ways and more emphasis on explaining why people think and behave in certain ways

 Involves smaller numbers of respondents

 Utilizes open-ended questionnaires or protocols

 Best used to answer how and why questions

 Not well suited to what, when and who questions Strengths Limitations

Complement and refine quantitative data Findings usually can not be generalized to the study population or community

Provide more detailed information to explain complex issues

More difficult to analyze; don’t fit neatly in standard categories

Multiple methods for gathering data on sensitive subjects

Data collection is usually time consuming and costly

How do you collect Qualitative Data? Observations

o Looking at what is happening rather than directly questioning participants o Used to better understand behaviors, their social context and meanings attached to them o Useful for certain populations - children, infants o Can identify unanticipated outcomes

Interviews (in-depth, individual)

o Usually provide rich data, details, insights from program participants and stakeholders about their experiences, behaviors and opinions

o Particularly useful for complex or sensitive subjects o Use open-ended questions

Focus Groups

o 8-12 people selected by non-random method, share some characteristics or experience relevant to the evaluation, ideally do not know each other, respond to questions from group facilitator

o Use group dynamics to generate data and insights

o Useful for generating ideas and strategies, defining problems in project implementation, assist with interpreting quantitative findings

o Open-ended questions or topics designed to stimulate discussion; topics usually broader than interview questions

These "Top Tips” were adapted primarily from: Evaluation for Learning: Basic Concepts and Practical Tools, LaFrance Associates, and User-Friendly Handbook for Mixed Method Evaluations, Division of Research, Evaluation and Communication of the National Science Foundation.