questionnaire design
lzy19951031
Week10-QuestionnaireDesignandAssign3-Part1-Expanded.pptxMarketing 743:
“Fundamentals of Marketing Research”
Week 10
Questionnaire Design
Week 10 - Lecture
“Questionnaire Design”
Learning Outcomes
3
After studying this lesson, you should be able to:
1. Know the key decisions in questionnaire design
2. Choose between open-ended and fixed-alternative questions
3. Avoid common mistakes in writing questionnaire items
4. Minimize problems with order bias
5. Understand principles of survey flow
6. Use latest survey technology to reduce respondent error
7. Appreciate the importance of pretesting survey instruments
Questionnaire Design
5
6
A Questionnaire
Is a standard set of questions that are asked of all those who participate in a survey.
6
7
A Questionnaire
Objectives are to make sure that:
We collect all information that we need to achieve our objectives;
We ask questions in the identical manner, no matter who the respondent is;
We ask the questions in the same sequence for all respondents; and
The data can be collated and analyzed with minimum confusion.
7
8
Steps in Writing a Questionnaire
8
Have class write down
Ensuring Quality
9
| Types of questions | ✓ | |
| Wording the questions | ✓ | |
| Programming instructions | ✓ | |
| Ordering the questions | ✓ | |
| Duration of the survey | ✓ | |
| Pre-screening the survey | ✓ |
Ensuring Quality
10
| Types of questions | ✓ | |
| Wording the questions | ✓ | |
| Programming instructions | ✓ | |
| Ordering the questions | ✓ | |
| Duration of the survey | ✓ | |
| Pre-screening the survey | ✓ |
Types of Questions
11
Types of
questions
Where: At the end, last questions of survey
Why: Make sure you can compare results across subgroups and provide interesting perspectives on the data.
Where: First questions of the survey
Why: Make sure that the respondents of the survey match the target population your client wants to reach.
Where: Middle (between screeners & profile)
Why: Make sure that the data you obtain answers the main research questions/objectives of the study.
Screening
questions
Profiling
questions
Objective-focused questions
Types of Questions: Screening
12
| My target is… | | So, my screening questions should be.. |
| All genders | | Please indicate whether you identify as… Male / Female / Gender minority (i.e trans, gender non-conforming, gender non-binary, etc |
| 18 years old or older | | Please indicate your age. 1-17 years old / 18-22 years old / 23-29 years old / 30-39 years old / 40 years old or older |
| Full-time students of Centennial College | | What is your current student status at Centennial College? Full-time student / Part-time student / Not a student |
| Users of the athletics and wellness centre | | How often do you train or work out at the athletics and wellness centre (AWC)? Every day / 3-5 times a week / 1-2 times a week / A few times a month / Once a month /Less often than once a month / I have never trained or worked out at the AWC |
Types of Questions: Objective Focused
13
| My goal is… | | So, some survey questions should be.. |
| To understand how the AWC can increase satisfaction among its current users and attract non-users? | | Q1. Have you ever trained or worked out at the athletics and wellness centre on the Progress campus? Yes / No Q2. Among users: Overall, how satisfied would you say that you are with the Athletics and Wellness Centre? Please use a scale of 1 to 10, where 1 indicates that you are very dissatisfied with it and where 10 indicates that you are very satisfied with it? Scale of 1 to 10 Q3. Among users: For each of the following aspects of the AWC, please assign the AWC a score of 1 to 10, where 1 indicates that the AWC performs very badly on this attribute and where 10 indicates that it performs excellently. Attribute 1 Attribute 2 Attribute 3 Q4. Among non-users: Why have you never trained or worked out at the athletics and wellness centre on the Progress campus? Open box |
Types of Questions: Profiling
14
| I also want to compare results across these subgroups… | | So, my profiling questions should be.. |
| Income | | Please indicate your household income. <$30 000 / $30 000-$49 999 / $50 000-$74 999 / $75 000 or more / Prefer not to answer |
| Frequency of exercise | | Generally speaking, how often do you exercise? Every day / 3-5 times a week / 1-2 times a week / A few times a month / Once a month /Less often than once a month / I never exercise |
| Preferred type of exercise | | Which of the following are your preferred methods of exercise? Please select all that apply. Examples: Boxing or martial arts / Swimming / Team sports / Weightlifting / Jogging / |
| Program studying | | Which of the following programs are you currently enrolled in at Centennial College? (list programs available) |
* Some profiling questions may have already been used as screening questions
Types of Questions: Profiling
15
- Consider what variables you want in the banner during analysis
- Make sure you have questions to create those variables
Making data mean more through storytelling | Ben Wellington | TEDxBroadway
16
https://www.youtube.com/watch?v=6xsvGYIxJok
17
Summary of Question Sequence
17
18
Summary of Question Sequence
In what order should we ask the questions?
Screening (Filtering) Questions
Identify appropriate respondents
Placed first
Warm-Up Questions
Use simple, interesting opening questions
E.g.: ask for an opinion
Answers may not be analyzed – intent is to build rapport, comfort and confidence
18
19
Summary of Question Sequence
Main Section
Contains the critical information we are looking for
Link back to objectives
Questions (in order)
Behaviour – what they do
Attitudes – what they think
Lifestyles – how they conduct their lives
19
20
Summary of Question Sequence
Classification Information
Demographic information - age, gender, income, education, and place of residence etc.
Always placed at the end of the questionnaire ... why?
20
Summary
21
Who is our target and how do we define them?
What are the main objectives of our study, and what questions do we need to ask in our survey in order to meet those objectives?
What subgroups would be interesting to compare the data between, and how do we establish those subgroups in our survey?
So, before starting your questionnaire, ask yourselves…
Ensuring Quality
22
| Types of questions | ✓ | |
| Wording the questions | ✓ | |
| Programming instructions | ✓ | |
| Ordering the questions | ✓ | |
| Duration of the survey | ✓ | |
| Pre-screening the survey | ✓ |
Question Wording: Considerations
23
Open-ended
Considerations
Closed-ended
Multiple
answer options?
Single-answer option?
What type of scale should be used?
(see lecture
from week 9)
List variety of options as answers and include an “other” in case you missed something
Answer option is always an “open box”
- Take less time
- Cheaper
- Easy to classify data
(no coding)
…but…
- Based on assumptions
- Don’t get “explanations”
- Provide explanation
- Discover surprises
…but…
- Have to code answers
- Detail is sparse
Question Wording
24
Clear, simple language
One idea per
answer option
Never assume or
have leading questions
Give clear instructions
25
Question Wording
How should we word the questions?
Goal: to avoid bias
Any factor that distorts/deviates from the true value
Deviates from the truth - value from sample is wrong
The wording must be clear
Does the question mean the same thing to all respondents?
25
26
Question Wording
Guidelines
Use ordinary words
Match vocabulary level of respondents
Be specific
Reword the following:
“How many times have you eaten out in the last month?”
How many movies have you watched in the past 30 days?”
26
27
Question Wording
Guidelines
Avoid complex questions / ideas that may be unfamiliar to respondents
Avoid abbreviations, slang, uncommon words
Example - car industry
ABS, AWD, EBA, FSI, VIN, MGFV
27
28
Question Wording
Guidelines
Consider the respondent’s ability to answer the question
Forgetfulness – keep referenced time periods short
“How many movies have you personally downloaded in the past year?”
“How much money did your household spend on groceries in the past year?”
28
29
Question Wording
Guidelines
Avoid double-barrelled questions
Deals with 2 issues / 2 questions linked together
“Would you say this product is expensive but good value for the money?”
Avoid ambiguous words
A word with more than one possible meaning
E.g. many, a lot
29
30
Online Exercise
List 5 ambiguous words that should not be used in framing questions.
30
Question Wording: Language
31
Clear, simple language
No double negatives:
Double negatives are when two “no”/”negative” words are used in the same sentence. This can cause confusion within your survey questions and result in the respondent answering in the opposite way that they intended to.
“To what extent do you disagree that students should not have to pay for post-secondary education?”
Strongly agree / Agree / Neither agree or disagree / Disagree / Strongly disagree
“How do you feel about the following statement: post-secondary education should be publicly funded by the government?”
Strongly agree / Agree / Neither agree or disagree / Disagree / Strongly disagree
Question Wording: Language
32
Clear, simple language
2) No jargon:
In every industry, specific terms and jargon are used. However, survey respondents are often not privy to this jargon, so only basic terminology should be used.
“To what extent did this advertisement match with ABC’s brand positioning?”
Very much / somewhat / not very much / not at all
“Was this ad consistent with your perceptions of ABC? “
Yes, very much / Yes, somewhat / No, not very much / No, not at all
Question Wording: Never Assume
33
Never assume
Don’t assume a respondent should be asked the question at all.
It is important that you filter the path respondents take so that they are only asked relevant questions. Assuming that a certain question is relevant to all respondents will force them to answer it, and this will lead to inaccurate data being obtained.
“At which restaurant on the Progress campus do you prefer to eat your lunch?”
-Assumes they eat their lunch at restaurants on campus.
First, ask a qualifying question,
“How often do you eat your lunch at a restaurant on the Progress campus?
Every day / a few times a week / Once a week / A few times a month / Less often than once a month / Very rarely or never
Question Wording: Never Assume
34
Never assume
2) Don’t make assumptions or lead in the actual phrasing of the question:
You may have hypotheses, but it is important that you ask questions in a neutral way and not introduce your own assumptions in the survey. This ensures that you end up understanding the true customer perspective, not just confirming your assumptions.
“Is poor customer service a reason why you do not eat lunch at Tim Hortons?”
Yes / No
“Do any of the following reasons explain why you do not eat lunch at Tim Hortons? Please select all that apply.”
attribute 1 / attribute 2 / attribute 3 / poor customer service
Question Wording: Instructions
35
Give clear instructions
Clarify the scales.
When using scales (1 to 5, 1 to 7, 1 to 10) always clarify what the extremes refer to. If no clarification is provided, respondents might answer according to their own assumptions, leading to unreliable data.
“Overall, how satisfied are you with the Athletics and Wellness Centre? Please use a scale of 1 to 10, where 1 indicates that you are very dissatisfied with it and where 10 indicates that you are very satisfied with it? “
Also, instead of using terms like “frequently”, clarify the actual times.
“How often do you eat your lunch at a restaurant on the Progress campus?
Every day / a few times a week / Once a week / A few times a month / Less often than once a month / Very rarely or never
Question Wording: Instructions
36
Give clear instructions
2) Clarify multiple choice answers vs. single choice ones.
When a variety of answer options are available for the respondent, it is important to communicate whether you want them to select the one answer option that best expresses their opinion, or whether you are allowing them to give a variety of reasons.
“Which of the following best explains why you do not train at the Athletics and Wellness centre? Please select one answer only.
reason 1 / reason 2 /reason 3 / reason 4
“Which of the following explain why you do not train at the Athletics and Wellness centre? Please select as many as apply”
reason 1 / reason 2 /reason 3 / reason 4
36
Question Wording: One Idea Per Answer
37
One idea per
question and answer
Target each question to one idea.
Tailor each question to address one point or idea. Trying to accomplish too much with one question will complicate the survey.
“Which of the following stores have you shopped at in the past month, and how much have you spent at each one?”
“Which of the following stores have you shopped at in the past month?
store 1 / store 2 / store 3
“Approximately how much have you spent at these stores in the past month?”
store 1 / store 2
Question Wording: One Idea Per Answer
38
One idea per
question and answer
Each answer option should focus on one idea
If one answer option focuses on multiple ideas, it will produce data that is difficult to interpret. Ensure that each answer option focuses on only one idea.
“To what extent do you agree with the following statements about brand xyz?”
Brand is cool / Brand is modern and innovative / Brand is classy and upscale
“To what extent do you agree with the following statements about brand xyz?”
Brand is cool / Brand is modern / Brand is innovative / Brand is classy /Brand is upscale
40
Facebook Survey
A & U
Establish participation
Perception
Changing Brand Opinion
The future of the Brand
Who do we need to talk too?
Ensuring Quality
41
| Types of questions | ✓ | |
| Wording the questions | ✓ | |
| Programming instructions | ✓ | |
| Ordering the questions | ✓ | |
| Duration of the survey | ✓ | |
| Pre-screening the survey | ✓ |
Programming Instructions
42
INPUT
logic
SKIP
logic
END
logic
Three types of instructions are available for the person who is…
programming the survey (if online)
reading the survey (if by phone)
administering the survey (if in-person)
Programming Instructions: End Logic
43
Applied in the screening questions of the survey
Ensures the sample matches the profile of the target population
End logic text: “Thank you for taking the time to respond to this survey. Unfortunately, we have already completed our quota of respondents who match your profile.”
| My target is… | | So, my screening questions should be.. |
| 18 years old or older | | Please indicate your age. 1-17 years old / 18-22 years old / 23-29 years old / 30-39 years old / 40 years old or older End if 1-17 years old |
| Full-time students of Centennial College | | What is your current student status at Centennial College? Full-time student / Part-time student / Not a student End if “part time student” or “not a student” |
| Users of the athletics and wellness centre | | How often do you train or workout at the athletics and wellness centre (AWC)? Every day / 3-5 times a week / 1-2 times a week / A few times a month / Once a month /Less often than once a month / I have never trained or worked out at the AWC End if “never trained or worked out at the AWC” |
Programming Instructions: Skip Logic
44
Applied anywhere in the survey skip logic is needed
Ensures that respondents only see questions that are relevant to their profile or how they answered previous questions
Skip logic text: “If answered (x) at Qx, skip to Qy”
| My goal is… | | So, some survey questions should be.. |
| To understand how the AWC can increase satisfaction among its current users? | | Q1. Have you ever trained or worked out at the athletics and wellness centre on the Progress campus? Yes / No If answered “no” at Q1, skip to Q4. Q2. Overall, how satisfied would you say that you are with the Athletics and Wellness Centre? Please use a scale of 1 to 10, where 1 indicates that you are very dissatisfied with it and where 10 indicates that you are very satisfied with it? Scale of 1 to 10 Q3. For each of the following aspects of the AWC, please assign the AWC a score of 1 to 10, where 1 indicates that the AWC performs very badly on this attribute and where 10 indicates that it performs excellently. Attribute 1 Attribute 2 Attribute 3 Q4. Why have you never trained or worked out at the athletics and wellness centre on the Progress campus? Open box |
Programming Instructions: Input Logic
45
Applied anywhere in the survey input logic is needed
Ensures that respondents can build on answers that they gave previously in the survey
| My goal is… | | So, some survey questions should be.. |
| To understand how the AWC can increase satisfaction among its current users and attract non-users? | | Q2. Overall, how satisfied would you say that you are with the Athletics and Wellness Centre? Please use a scale of 1 to 10, where 1 indicates that you are very dissatisfied with it and where 10 indicates that you are very satisfied with it? Scale of 1 to 10 Q3. Why did you give a score of (input score from Q2) to indicate your level of satisfaction with the Athletics and Wellness centre? Open box |
Ensuring Quality
46
| Types of questions | ✓ | |
| Wording the questions | ✓ | |
| Programming instructions | ✓ | |
| Ordering the questions | ✓ | |
| Duration of the survey | ✓ | |
| Pre-screening the survey | ✓ |
General Flow of the Questionnaire
47
Broad questions
(general usage habits, frequency of buying, reasons for buying)
Specific questions
(brand attitudes, levels of satisfaction with a product or service)
Survey questions should get more specific as they advance.
This flow helps avoid introducing any bias in the study,
as respondents don’t realize what the focus is until the end
48
Summary of Question Sequence
Funnelling
Starts with generalized questions and proceeds to more and more specific questions
48
49
Summary of Question Sequence
Wherever possible:
Place easier questions before difficult ones;
Place less personal questions before the more personal and sensitive ones; and
Factual questions should appear before ones that require judgment.
49
Questionnaire – Leading?
50
Yes Minister – Question Ordering design
https://www.youtube.com/watch?v=G0ZZJXw4MTA
What did you think? – funny but true!
Open vs. Close-ended Questions
LO4
51
52
Open-ended Questions in Surveys
Advantages
Rich information in the consumer’s own language
Use to interpret close ended questions
Can suggest other alternatives not listed
52
53
Open-ended Questions in Surveys
Disadvantages
Bias towards articulate respondents?
Interviewer bias (recording shortcuts)
With self-administered questionnaires (no interviewer to probe): shallow, unclear answers?
Editing and coding (i.e. preparation for analysis) can be time consuming
Too many categories result in non- robust data patterns
53
54
Probes
Interviewer encourages the respondent to elaborate or clarify a response
“What do you mean by ….?”
“Give me an example of ….”
54
See examples p
55
Dichotomous
Choose between 2 answers
Examples
The federal government does care what people like me think.
Agree / disagree
Toronto should make the Gardiner Expressway a toll road
Agree / disagree
Close-ended Questions
55
56
Multiple-choice
Choose among several answers
Example
How old are you
Less than 20
20 – 29
30 – 39
40 – 49
50 – 59
60 or over
Close-ended Questions
56
Open Ended / Close Ended
57
This means that you get their opinions (spontaneous feedback)
before you introduce any potential answer options (aided feedback)
Open ended questions should be before closed ended
(unless you are are asking for an explanation)
For what reasons do you not train or work out at the Athletics and Wellness Centre on the Progress campus?
(Open box)
Which of the following explain why you do not train or work out at the Athletics and Wellness centre? Please select as many as apply”
reason 1 / reason 2 /reason 3 / reason 4
Randomizing Answer Order
58
This ensures that each respondent sees answers in a different order.
In turn, results at a given question are not affected
by the order that you happened to list them in.
When multiple answer options are provided,
it is always better to randomize order
Which of the following explain why you do not train or work out at the Athletics and Wellness centre? Please select as many as apply”
Instructions to programmer: randomize order
reason 1
reason 3
reason 4
reason 2
Randomizing Stimuli
59
This ensures that the reactions to the stimuli or ideas are
not affected by the order in which they were presented.
When testing advertising materials,
or when you are testing potential changes, randomize order.
The following is a billboard ad that could be displayed on campus for the Athletics and Wellness centre. How much do you like this ad overall? Please use a scale of 1 to 10, where 1 indicates that you really dislike the ad and where 10 indicates that you really like it.
Version 1
The following is a billboard ad that could be displayed on campus for the Athletics and Wellness centre. How much do you like this ad overall? Please use a scale of 1 to 10, where 1 indicates that you really dislike the ad and where 10 indicates that you really like it.
Version 2
Instructions to programmer: Randomize order so half of the respondents see Version 1 first and the other half see Version 2 first.
Ensuring Quality
60
| Types of questions | ✓ | |
| Wording the questions | ✓ | |
| Programming instructions | ✓ | |
| Ordering the questions | ✓ | |
| Duration of the survey | ✓ | |
| Pre-screening the survey | ✓ |
Duration of The Survey
61
Why have estimates?
Respondents are promised incentives based on estimated time of survey
(the longer the survey, the greater the incentive should be)
Important to know ahead of time how long the survey will be
(if survey is longer than anticipated, incentive will seem too low,
and you won’t attain your sample size)
Try to keep on-site surveys to 10 minutes at most
How do you establish estimates?
When estimating the time, assume 5-6 seconds per basic profiling question
(age, gender, income, place of residence, etc)
When estimating the time, assume 25 seconds per general question
(satisfaction, key criteria, etc)
Best way to have a clear idea of timing…prescreen!
Ensuring Quality
62
| Types of questions | ✓ | |
| Wording the questions | ✓ | |
| Programming instructions | ✓ | |
| Ordering the questions | ✓ | |
| Duration of the survey | ✓ | |
| Pre-screening the survey | ✓ |
Pre-Screening
63
Tests for any lack of clarity in the wording of questions
Ensures that the order of the questions is logical
Determines if the length of the survey is in line with your estimate
Tabulating data from prescreens will highlight any issues
Conduct prescreening sessions with 2-3 members of the target population
Assignment 3 – Questionnaire Design DUE – Week 11 – Monday 3rd August at Noon 10% of Total Grade
Questionnaire Design Assignment
66
Individual or Pairs 1-2 people maximum
Select a business problem from the attached list, and describe the main research question that has to addressed in the research study – the main research objective
Draft a 2 - 3 page questionnaire (that could be used in an online study), whose findings would answer the main research question – remember to include the screening questions, the objective focused questions and the profiling questions
To be prepared on an individual or pair basis (each member will be assigned the same grade)
Remember to include a mix of open ended and closed questions
Due by noon Monday 3rd August – our last assignment before the final exam!
Questionnaire Design - Topics
67
Choose one from the following list of topics for the questionnaire design:
.The Premiere of Ontario – Doug Ford – has talked about changing the idea of the LCBO selling Cannabis and making it open to private business. How would you help Ford decide what the people of Ontario want and how Cannabis should be distributed.
How can we measure the impact of recent press releases around privacy of information with regard to Facebook and how it has potentially impacted the brand image of Facebook.
Do Ontarians believe in Global Warming? True or false? How can we measure the level of understanding and support for the concept?
Gun violence in Ontario – how to address it, who is most afraid, what are the underlying causes and how could they be addressed? (Focus on one or two ideas – not all have to be included)
How has Covid 19 effected your life?
