Creative Exercise #8
Data Collection
Part 1 – Survey
1
Survey Research
Survey
Data-collection technique in which information is gathered from respondents by having them respond to questions or statements.
Characteristics
Data is collected from large samples of people.
Respondents presented with questions.
Measure what people say, not what they do.
The term survey both designates a specific way of collecting data and identifies a broad research strategy. Survey data collection involves gathering information from individuals, called respondents, by having them respond to questions. We use survey to gather data as part of many of the research methods discussed in previous chapters, such as qualitative studies, quantitative studies, experiments, field research, and evaluations. In fact, the survey probably is the most widely used means of gathering data in social science research.
Survey typically involve collecting data from large samples of people; therefore, they are ideal for obtaining data that are representative of pupulations too large to deal with by other methods.
Second, all surveys involve presenting responsdents with a series of questions to answer. These questions may tap matters of fact, attitudues, opinions, or future expectations. The questions may be simple, single item measures or complex like multiple item scales. Whatever the form, however, survey data basically are what people say to the investigator in response to a question.
it is important to know that survey measures what people say about their thoughts, feelings, and behaviors, not their actual thoughts, feelings, or behaviors. So, whenever we conduct a research using survey technique, there exist an underline assumption about the limitation of this method.
2
Two Basic Forms of Survey
We collect data in survey research in two basic ways: with a questionnaire or with interviews.
First, the questionnaire contains recorded questions that people respond to directly on the questionnaire form itself, without the help of an interviewer. A questionnaire can be handed directly to a reponsdent: mailed or sent online to the members of a sample, who then fill it out on their own and send it back to the researcher; or presented via computer.
An interview involves an interviewer reading questions to a respondent and then recording his or her answers. Researchers can conduct interviews either in person or over the telephone. This is more common in situations when your repondents may need assistance due to literacy, or lack of physiological functional ability to read and comprehend the questions on their own. EX)
Sometimes, survey research can use both questionnaire and interview techniques, with respondents filling in some answers themselves and being asked other questions by interviewers.
3
A questionnaire contains written questions that people respond to directly on the questionnaire form itself, without the aid of an interviewer.
An interview involves an interviewer reading questions to respondents and recording their answers.
Designing Questions
Closed-Ended
Fixed set of alternatives
All possible theoretically relevant options determined in advance
Ease of data handling
Open-Ended
Respondents develop own responses
Response options are not predictable
Complex data handling
Respondent may give multiple answers
Closed ended questions provide respondents with a fixed set of alternatives from which to choose. The response formats of multiple choice is an example of closed ended question.
All possible relevant options should be available for respondents to choose because if they have something else in mind as an answer but there is no category or item for them to choose that represents their thoughts or answer, it will be problematic. But, this is hard and that is one of the cons of closed ended question. In a closed ended question, you can not anticipate to find all answers because humans are hard to predict.
Closed ended questions are relatively easier to handle in terms of analyzing the data because it can be categorized and numbered. We will learn more about it in our later chapters when we learn to analyze the data.
One easy example of this can be when I try to analyze your exam scores and mid evaluations. Having your exam scores, I can easily analyze to see the average score, top scores, lowest score, median and and so on, whereas when I collected the mid evaluation from all of you which the questions were in open-ended form, I had to read each one of your answers, and write a note about it. Look for any overlapping opinions or anything that I find it important.
4
Dichotomous
Multiple-choice
Likert-scale
Question response options
Forced ranking
Adjective checklist
Smiley face continuum
Wording of Questions
Because the questions that make up a survey are the basic data gathering devices, researchers need to word them with great care. Especially with questionnaires that allow the respondent no opportunity to clarify questions, not being clear can cause substantial trouble.
You need to avoid anything bias, manipulating, or confusing. Leading question is an example of a biased question. If I ask how dumb is our park district when it comes to renovation? This immediately brings a negative connotation to the question. Instead you should ask like please describe your opinion about your park district’s renovation policies.
Other than special cases when you want to ask about the past experiences or future expectations, you should always use the present tense. Consistency is important and you should always stay consistent using the present tense in all your questions.
Keeping it simple and direct is important! Your reponsdents are probably going to be answering more than couple questions, so it needs to be as simple and direct as it possiblely can, otherwise it will make them tired and also give them misunderstanding. For example, imagine if you were asked: How happy or unhappy are you with the rate of current park district funding and the its hours of operations? What is it that you are trying to ask right? What about this question, this is a real example that was used in one of the survey about the Holocaust, “does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” right? Confusing.
This is not the time to show off your vocab skills. It is often recommended that the questions should be written in a level when a fourth, fifth graders can understand what it means.
For example, in a questions that says, children who get into trouble typically have had a bad home life: is an undersirable statement because it includes two sources of vagueness. The word trouble is unclear, what kind of trouble? Trouble with the law? With the school? With parents? Also what about the phrase bad home life? What is considered a bad home life?
Academic jargon: transcend, discourse, liminal, heteronormative, etc.
Slangs: kinda, wanna, cheesy (=cheap or tacky), kush, 420, pot, grass, mary jane or whatever
8
Avoid bias/manipulation/confusion
State questions in present tense
Simple and direct is best
Low reading ability of some respondents
Avoid reference to vague terms
e.g. “trouble,” “bad home life”
Avoid academic jargon
Avoid slang
Pretest
Structure and Design
Provide precise directions
Order questions:
Early questions should not bias later ones
Increase interest in completing instrument
Question formats:
Straightforward and convenient
for respondents
It is important to provide precise directions. First thing they will read before question is the direction and if that makes people confusing, it is going to negatively impact the respondents throughout the entire survey questions. Besides the leaset thing you want to do is recruit people to participate in a survey and have them misunderstand something so they completely provide wrong information or decides to not finish the survey.
Order questions:
Generally, questions that are asked early in the questionnaire should not bias answers to those questions that come later. For example, if we asked several factual questions regarding poverty and the conditions of the poor, and later we asked a question concerning which social problems people consider serious, more repondents will likely to include poverty than otherwise would have done so. When questionnaire contains both factual and opinion questions , we sometimes can avoid these potentially biasing effects by placing opinion questions first.
It is also important to design your questions so that interesting subjects can be placed first.
9
Response Rate
Response rate: the proportion of a sample that completes the questionnaire or interview.
Non-respondents may differ in significant ways from responders.
Data from obtained sample may no longer be representative of the population.
Checking for Non-response bias
Compare characteristics of sample obtained to known characteristics of the population.
10
Ex) If my population was college students and my sample is RST undergrad students, and if mostly the seniors and juniors responded whereas freshman or sophomores did not, it no longer represents the population
10
How to increase response rate?
Cover Letters
Introduction, importance, anonymity
Incentives
Payment, gift cards, stamped self-addressed envelope
Follow-Ups
Two follow-ups can generate 15-20% increase in response rate
Step 1: Reminder letter
Step 2: Letter + additional questionnaire
As length increases response rate declines
Avoid crammed appearance
A properly constructed cover letter can help increase the response rate. It introduces what the purpose of the study and what will be asked and how, but also talks about why this is important to persuade the participants. A well documented cover letter also provides credibility of the survey as well. Also, cover letter explains detail about the survey. For instance, how the anonymity will be kept and non of the personal information will ever be exposed under any circumstances can be explained in the cover letter.
11
Assessment of Questionnaires
Faster and cheaper than interviews
Able to reach geographically dispersed sample
Effective with sensitive topics
Eliminates problem of interviewer bias
Require some literacy
Written instructions
Can’t probe
Can’t control who completes it
Responses not independent
High response bias
12
Summary: Types of Surveys
13
Data Collection
Part 2 – Interview & Focus group
14
Interview Structure
During an interview, the investigator or an assistant would read the questions directly to the responsdents and then records their answers. Interviews offer the investigator a degree of flexibility that is not available with questionnaires. One area of increased flexibility relates to the degree of structure built into an interview.
The element of structure in interviews refers to the degree of freedom that the interviewer has in conducting the interview and that respondents have in answering questions. We classify interviews in terms of three levels of structure:
Unstandardized which is unstructured and nonscheduled standardized which is equal to semi structured, and schedule standardized same as structured.
15
Unstructured - Unstandardized
Semi structured - Nonschedule-standardized
Structured - Schedule-standardized
Interview Structure
Unstructured - Unstandardized
Interviewer develops own questions and probes as interview progresses
Semi structured - Nonschedule-standardized
More narrow topic
Specific questions asked of all respondents
Interviewer free to probe and rephrase
Structured - Schedule-standardized
Specific instructions for interviewer
Specific questions in fixed order and transition phrases for interviewer
As you could easily guess, unstandardized has the least structure. All the interviewer typically has what is called interview guide which would list of questions that they aim to ask. However during the unstructured interview, interviewer would not rely on the interview guide, rather he/she would develop questions as their conversation evolves. This type of interview heavily rely on the skills of the interviewer to ask good questions and keep the interview going, so this technique is recommended only when there is an expert in terms of conducting an interview exists. This unstructured approach is especially appropriate for exploratory research because relatively little is known about the concept and there exists few knowledge that you can ask about, so want to focus on whatever that comes up from the respondents.
16
Interview Structure
Unstructured - Unstandardized
Interviewer develops own questions and probes as interview progresses
Semi structured - Nonschedule-standardized
More narrow topic
Specific questions asked of all respondents
Interviewer free to probe and rephrase
Structured - Schedule-standardized
Specific instructions for interviewer
Specific questions in fixed order and transition phrases for interviewer
Nonschedule-standardized interviews add more structure than the previous one. It is often used when asking about the narrower topic and specific questions to respondents. The interview still remins farly conversational, the interviewer is free to probe if necessary and rephrasing questions is also okay. Also, interviewer can change the order of questions depending on the situation. So, literally it is somewhat similar to the unstructured, but there exits some questions that you make sure to ask all respondents no matter what. So, if you are wanting to ask questions about stress, you can freely have conversation about it, but in a semi structured, if you aim to know more about eustress which is a positive stress, you will ask all participants some questions about the eustress.
17
Interview Structure
Unstructured - Unstandardized
Interviewer develops own questions and probes as interview progresses
Semi structured - Nonschedule-standardized
More narrow topic
Specific questions asked of all respondents
Interviewer free to probe and rephrase
Structured - Schedule-standardized
Specific instructions for interviewer
Specific questions in fixed order and transition phrases for interviewer
Lastly, we have schedule standardized which is the most structured type. Specific instructions exists for the interviewer to follow, with specific questions to be asked in a fixed order. And transition phrases for the interviewer to use. Very well documented and worded script or interview guide exist in this case, you interviewer must be strict in following those guide. It is almost like you as an interviewer reading the scripts as respondents answers to your question. This is good that it makes certain that all questions you aim to ask are being asked, however, it gives less room for the participants voice to be considered or heard as the conversation continues.
18
Structure
Degrees of structure
Semi structured interviews (or other data collection tool of this sort)
Various degrees of structure
Surveys
Deductive approach
No/minimum structure
Maximum structure
In-depth open-ended interviews
Inductive approach
19
Conducting an Interview
Start with simple questions
Presence of third parties
Probes: follow-up questions intended to elicit clearer and more complete responses
Classifying responses and summarizing the "high points"
Taking verbatim notes
Making an audio or video recording of the interview
Things to consider …
20
The interview is a social relationship designed to exchange information between the respondent and the interviewer. The quantity and quality of information exchanged depend on how astute and creative the interviewer is at understanding and managing that relationship.
When beginning an interview, it is recommended to start with fairly simple and nonthreatening questions because otherwise, interviewee may feel the pressure and the entire conversation may be disturbed or negatively influenced. It is like how we have a introductory time in the very beginning of the course. Imagine if I give you guys a full lecture on measurement or populations and samples in the very first day, many of you may end up dropping the course right?
Sometime, depending on the situation, a third person may be present during the interview whether that is someone related to the interviewee or one of the assistant researcher from our research team to assist an interview. Utilizing the resource is important, so if you ever can get an assistant to be present at the interview that person can take notes. Taking verbatim notes is important during interview because every single word they say could mean something. Also, it is important to take notes of their moods, facial expressions, anything that you can detect from the conversations, because that means something and also that can’t be noticed when you just hear the recorded audio file.
20
Assessment of Interviews
Flexibility
Can motivate respondents
Able to explain items to respondent
Add observational data
Cost
Time
Interviewer bias
Variation in meaning between interviewers
21
Focus Groups
Focus group (group depth interview)
is an interview with a whole group of people at the same time
Led by one or more moderators
Moderator(s) direct the discussion by following an outline of the main topics of inquiry
When you need to gather ideas, when you think it would better encourage active conversation between people, not too sensitive or personal because people would then not share in front of others.
22
23
Focus Groups
More flexible than questionnaire
Less-expensive and faster than individual interviews
Less generalizable than survey
Difficult to analyze
Qualitative rather than quantitative data
24
Key Ideas
No perfect method
The choice is a matter of fit to the research problem and population
Surveys
Questionnaires and [semi] structured interviews
Closed-ended vs. open-ended questions
Clear instructions and questions’ order (order effect)
Response rate
Cover letter, incentives, length
Focus groups – group dynamic as a source
Generating ideas
Getting feedback
25
image1.jpg
image2.png
image3.svg
.MsftOfcThm_Accent1_Fill { fill:#4472C4; } .MsftOfcThm_Accent1_Stroke { stroke:#4472C4; }