what could have gone wrong
ddhaocc
session_7_8_9_survey.pdf
Survey B U S 1 1 5 W I N T E R 2 0 1 7
Survey — the overwhelming choice for primary data
2
Definition Interviews with a large number of respondents using a predesigned questionnaire.
◦ Standardized yet somewhat flexible
◦ Relatively easy to administrate
◦ In some cases, respondent is the best source for information
◦ Considerable judgment: sampling, questionnaire design.
◦ Good analytical skills required
Features
Four Basic Types of Survey
3
No computer Computer
Interviewer
Person-administered surveys ◦ Interviewer reads questions either face-to-
face or over the telephone
Computer-assisted surveys ◦ Person-administered with computer
assistance (e.g. question randomization, skipping rules, etc.)
No interviewer
Self-administered surveys ◦ The respondent completes the survey on
his or her own on paper-pencil
Computer-administered surveys ◦ The respondent completes the survey on
his or her own on computer
Interviewer (vs. self) ◦ Advantages:
◦ Quality control
◦ Progress control
◦ Allows feedback
◦ Reduce respondents’ workload
◦ Disadvantages:
◦ High cost
◦ Sensitive questions/social desirability
4
Computer (vs. paper) ◦ Advantages:
◦ Quality control
◦ Flexibility: multi-media, different (fancy) measurement
◦ Reduce data entry cost
◦ Disadvantages:
◦ Requires computer knowledge and (in most cases) internet connection
5
Recruiting and Survey Methods ◦ In-home survey
◦ Intercept survey (mall, street, campus)
◦ Telephone survey
◦ Mail survey
◦ Email/Online survey
◦ Qualtrics
◦ Survey Monkey
◦ Google Consumer Survey
6
Questionnaire Organization
7
Guidelines of Questionnaire Organization
8
◦ Begin with an introductory paragraph:
◦ The purpose of the survey
◦ Who is running the survey
◦ How the survey data will be used
◦ Whether the surveys will be anonymous
We are interested in learning more about how consumers respond to different situations. In this study, you will be asked to evaluate a situation as though it is real. You will then be asked about your response to this situation.
At times you will feel as though you don’t have enough information. Please do your best to respond with the information that you do have. There are no right or wrong answers – it is your unique responses that we are interested in.
Thank you in advance for your help with our research!
Guidelines of Questionnaire Organization
9
◦ Open with easy and non-threatening questions.
◦ Questionnaire should flow smoothly & logically from one topic to next.
◦ Group similar question together. Don’t jump around excessively.
◦ Generally, better to go from broad (general) questions to specific questions.
◦ Check MR2 8-1f for more useful guidelines Rate the brands on these attributes.
What types of beverages do you drink on a weekly basis?
Which of the following beverages is your favorite?
How often per week do you drink this beverage?
In what occasions do you drink this beverage?
Guidelines of Question Wording ◦ Ask applicable questions
■ Did you find it useful when you call our 800 service number?
◦ Use simple words
■ “Is the distribution of snack foods adequate?” (X) ■“Are snack foods readily available when you want to buy them?” (✓)
◦ Avoid “double barreled” questions
■ “Are you satisfied with the food quality and service of our brand”? (X)
◦ Avoid leading or biasing questions
■ “Colgate has the largest market share among all toothpastes, is it your favorite toothpaste brand?” (X)
10
Guidelines of Question Wording
11
Principle
Always remember that survey is a conversation. If the questions doesn’t sound right to speak out, it’s probably not a good question.
Measurement and Scale
12
What is measurement?
13
Definition The process of turning concepts into numbers following specific rules.
◦ Nominal (Identity): ◦ What’s your ethnicity? White Black Asian Latino Native American
◦ Ordinal: Answers can be rank-ordered ◦ What’s your income? 10,000 or below 10,000-50,000 50,000-100,000 100,000 or above
◦ Interval: Answers can be rank-ordered + equal distance between answers ◦ On a scale of 1-10, what is your pain level? ______
◦ Ratio (absolute magnitude): Answers can be rank-ordered + equal distance between answers + zero means “nothing”
◦ How much are you willing to pay for this product? $ ______
14
Lowest
Highest
Different types of scales
How much do you like apple juice?
Not at all 1 2 3 4 5 6 7 Very much
How much do you like orange juice?
Not at all 1 2 3 4 5 6 7 Very much
How much do you like apple juice?
Not at all 0 1 2 3 4 5 6 Very much
How much do you like orange juice?
Not at all 0 1 2 3 4 5 6 Very much
“Levels” of Measurement
15
How much do you like apple juice?
Not at all 0 1 2 3 4 5 6 Very much
How much do you like orange juice?
Not at all 0 1 2 3 4 5 6 Very much
“Levels” of Measurement
16
How many bottles of apple juice do you drink every month?
Not at all 0 1 2 3 4 5 6+
How many bottles of orange juice do you drink every month?
Not at all 0 1 2 3 4 5 6+
“Levels” of Measurement
17
◦ A same variable can usually be measured in different levels of
measurements
◦ Soft drink (MR2 Exhibit 7.2)
◦ Age
◦ Always use the highest level of measurement available
◦ Because higher level measurement can be converted to lower level but
not the other way around.
Itemized scales (MR2 7-2a)
18
To what extent do you agree with the following statement?
A scale on which respondents express their STRENGTH OF AGREEMENT with each of several statements
Likert Scale
Measurement Problems
19
Open- vs. Closed-ended
20
a. “Please write down any thoughts or feelings you had while viewing the advertisement.”
_________________________________________________
b. “If you were to purchase a new automobile this year, what brand of automobile would you most likely purchase?”
_________________________________________________
c. “Based on what you saw and heard in the commercial, which of the following words or phrases best describes the kind of person you feel would be most likely to use this product?” (check all that apply)
____Young ____Home-owner ____Single ____Old ____Modern ____Career woman
d. “If you were to purchase a new automobile this year, which one of the following automobiles would you most likely purchase?”
____Toyota Camry ____Honda Accord ____Nissan Maxima
Open- vs. Closed-ended
21
◦ Disadvantages: ◦ Respondents vary significantly in ability to provide clear responses ◦ Respondents vary significantly in willingness to compose written answers ◦ More effort for the interviewer and the data analyst ◦ Misses the answer that respondents couldn’t easily think of ◦ Unexpected ambiguity leads to unexpected answers
◦ Advantages: refer to disadvantage of the closed-ended questions
Open-ended
“The answer that respondents couldn’t easily think of”
22
◦ “What is the most important thing that children should learn to prepare them for life?”
◦ “To think for themselves” ◦ 5% mention this in an open response format ◦ 62% endorse it in a closed response format
Schuman & Presser (1981)
Open- vs. Closed-ended
23
◦ Disadvantages: ◦ Sometimes there are too many options to list ◦ Good questions (and responses) are hard to develop ◦ Measurement and scale — choice of measures and analysis methods ◦ Provides fewer opportunities for self explanation ◦ Hard to detect respondents’ ignorance ◦ Unable to detect the salience of an option
◦ Advantages: refer to disadvantage of the open-ended questions
Closed-ended
Open- vs. Closed-ended
24
Purpose: To find out how many people consider power to be the most important factor when buying a car.
1. What do you think is the most important factor? __________
2. Which of the following do you consider to be the most important factor?
A. Power B. Safety C. Price D. MPG
3. Do you consider power to be the most important factor?
___ Yes ___ No
Example: How to avoid (create) biases
Measurement Problems
Bias vs. Error or (lack of) Validity vs. Reliability or (lack of) Accuracy vs. Precision
25
No Bias (Valid)
Bias (Not Valid)
Low Error (Reliable)
High Error (Unreliable)
26
How to reduce error?
27
◦ Use the right sampling frame (to access the right respondents). ◦ Avoid ambiguity ◦ Reduce the impact of uncertainty and respondent ignorance. ◦ Use more than one item to measure different aspects of the same concept.
- What did you learn in BUS 115 to reduce error? - I learned that using the appropriate question form can reduce error.
vs. - I learned that “using the appropriate question form can reduce error.”
28
◦ Wording ◦ "Church preference?" ◦ "Sex?"
◦ Context ◦ "What have you done today?" ◦ "Where do you live?" ◦ “Why do you use crest?” – doctor’s recommendation vs. fighting cavity
Ambiguity
Ambiguity
29
• 1. How often did you go shopping last week? • 2. How much did you spend on a typical occasion?
$8.23
• 1. How often did you go shopping last month? • 2. How much did you spend on a typical occasion?
$32.14
• 1. How often did you go shopping last semester? • 2. How much did you spend on a typical occasion?
$53.12
Ambiguity
30
Did you do any work for pay last week?
Students Employees
Donating blood for $50 70% 20%
Mowing neighbor‘s lawn 100% 70%
Baby sitting 100% 50%
Schwarz (1987)
Ambiguity
31
• How frequently do you have headaches?
Never Occasionally Sometimes Frequently
• How frequently do you have heart attacks?
Never Occasionally Sometimes Frequently
Uncertainty and Respondent Ignorance
32
◦ If many respondents are likely to be uncertain, provide a neutral response. Based on what you saw and heard in this commercial, how interested do you feel you would be in buying the product? Definitely won’t Probably won’t May or may not Probably would Definitely would
◦ If many respondents are likely to be ignorant (lacking knowledge) of the topic, provide a “don’t know” category.
Is your TV News program... Showing a profit Breaking even Showing a loss Don’t know
33
Loyalty to a Given Brand
“This product category is very important to me.”
"I would be willing to pay a higher price for this brand over others." "I will buy this brand
the next time I buy.”
“If I like a brand, I rarely switch from it just to try something different.”
"I am committed to this brand."
"I intend to keep purchasing this brand." ?
Multiple Measures
Multiple Measures: Cronbach’s Alpha
34
SPSS: Analyze=> Scale => Reliability Analysis…
How to reduce (create) bias?
35
◦ Use the right sampling frame (to access the right respondents). ◦ Use the appropriate question form (open vs. closed ended). ◦ Be aware of the order effects. ◦ Be aware of the scale effects.
Order effects: Question Order
36Strack, Martin, & Schwarz (1988)
1. What is your dating frequency?
2. How satisfied are you about your life?
r = .67
1. How satisfied are you about your life?
2. What is your dating frequency?
r = -.11
Order effects: Question Order
37
Do you think the U.S. should let Communist reporters come in here and send back to their papers the news as they see it? 45% said No
Do you think a Communist country should let American newspaper reporters come in and send back to their papers the news as they see it? Most said Yes
25% said No
Order effects: Question Order
38
Do you think it should be possible for a pregnant woman to obtain a legal abortion if she is married and does not want any more children? 69% said Yes
Do you think it should be possible for a pregnant woman to obtain a legal abortion if there is a strong chance of serious defect in the baby? Most said Yes
50% said Yes
Schuman & Presser (1981)
Order effects: Framing
39
“Would you say that your female teachers were more empathetic with regard to academic and personal problems than your male teachers, or were they less empathetic?”
41% said females more empathetic, 12% said males more empathetic.
Wänke, Schwarz, & Noelle-Neumann (1995)
“Would you say that your male teachers were more empathetic with regard to academic and personal problems than your female teachers, or were they less empathetic?”
9% said females more empathetic, 55% said males more empathetic.
Order effects: Option Order
40
◦ Answer depends on which alternative is “tested” first: ◦ Visual mode: first alternative, resulting in primacy effect ◦ Auditory mode: last alternative, resulting in recency effect. ◦ Discrepancy is larger among elderlies.
Krosnick & Alwin (1987)
“Which gender of your teachers was more empathetic with regard to academic and personal problems?”
A. Male B. Female
Scale Effects
41
37.5%
How many hours do you watch TV everyday?
Schwarz et al. (1985)
Scale Effects
42
Not so successful 0 1 2 3 4 5 6 7 8 9 10 Extremely successful
Not so successful -5 -4 -3 -2 -1 0 1 2 3 4 5 Extremely successful
“How successful would you say you have been in life?”
34%
13%
Schwarz et al. (1991)
Data Quality Issues
Cheating ◦ Cheating: respondents/interviewers
◦ Faking response in a survey
◦ Solution: add validation questions; eyeballing (raw data inspection)
44
Nonresponse
◦ Response rate = completion/total requests
◦ Important?
◦ OK if for generic reasons
◦ Bad if nonresponse due to their potential answer
◦ Example: lunch calorie intake
◦ Nonresponse rate: male — 10%; female — 25%
◦ What if the study is about lunch cost?
◦ Solution: incentive; pre-commitment; good introduction/persuasion skills (face-to-face)
45
Nonresponse to the request
Nonresponse
◦ Don’t Know
◦ Solution: provide a “don’t know” option
◦ Unintentionally omit
◦ Solution: force response (computer administrated); use an interviewer; imputation — using other information to predict the missing information
◦ Sensitive questions
46
Nonresponse to certain questions
Sensitive Questions
47
Female Male
Past year 1.7 4.5
Past 5 years 3.9 14.7
Lifetime 6.5 22.8
Tourangeau et al. (1997)
Self-reported Number of Sexual Partners
Types of Sensitive Questions ◦ Personal information: income; identifiers
◦ Sex related
◦ Socially desirable/undesirable behavior
◦ Do you avoid certain foods because they are high in fat or cholesterol?
◦ Do you almost always wear a seat belt when you drive or ride in a car?
◦ How many times do you work out every week?
◦ Do you have a library card?
◦ Did you vote in last election?
48
Solutions (MR2 Exhibit 8.2) ◦ Assure anonymity/confidentiality
◦ Avoid using an interviewer
◦ Using non-precise scales (e.g. income)
Below $30,000
$30,000 - $50,000
$50,000 - $100,000
$100,000+
◦ Ask sensitive questions last if possible
◦ Manage the contextual questions
◦ Randomize the question each participant answers
49
Solution: Manage the contextual questions ◦ Embed question within related set of questions
Q1. “How many soft drinks do you drink per week?”
Q2. “How many dairy drinks do you drink per week?”
Q3. “How many alcoholic drinks do you drink per week?”
◦ To communicate that the behavior is normal
Q1. In general, would you say that you drink more than your friends, less than your friends, or about the same amount as your friends?
Q2. Think about the friend you know who drinks the most. About how many drinks would you say that person usually has?
Q3. And how about you? On days when you have any alcoholic beverages, about how many drinks do you have?
50
Solution: Randomize the question ◦ Limitation: only obtain population estimate
◦ Example: Suppose Durex want to learn how many condoms on average college students use every week...
◦ Which question each respondent answer is decided by a coin toss (managed by themselves): Head – Question A; Tail – Question B
◦ Question A: How many condoms do you use every week?
◦ Question B: What day of the week is your birthday this year? Sun = 0, Mon = 1, Tue = 2, Wed = 3, …, Sat = 6
◦ If the average of total respondents is 2.8, what’s the average number of condom use?
◦ Base on the coin toss, half should answer A and the other half should answer B. For the half who answered B, what should their average answer be?
◦ 3.
◦ And you want to know the average of the other half (that is, who answered A).
◦ The average of all participants is 2.8. The average of half of them is 3, what is the average of the other half?
51
