Marketing Research V Power Point

8 | FEBRUARY 2015 | TRAINING & DEVELOPMENT WWW.AITD.COM.AU

INTELLIGENT QUESTIONNAIRE DESIGN FOR EFFECTIVE PARTICIPANT EVALUATIONS BY LISA ELIAS

E valuating learning and development programs is about much more than just putting together random

questionnaires, distributing and collecting them at the end of a workshop. Surveys are a form of social research and, conducted effectively as a holistic process, can provide a powerful opportunity to elicit valuable data from a captive, immediate audience. When analysed and reported, the results can be used to celebrate and communicate successes and/or to facilitate continuous improvement on a range of levels.

The vast majority of learning and development programs, courses and workshops are “evaluated” using traditional participant feedback forms. These are usually (although not always) paper-based, and are variously called questionnaires, surveys, reactionnaires or happy sheets. They are often mistakenly understood to be meeting “Level 1” requirements with reference to the Kirkpatrick model.

Designing a questionnaire So how can we do this better? A well- designed questionnaire is a key element of the evaluation process. Here are four common steps to good design:

1 Identify the objectives Before designing your survey it is essential that you first identify your objectives – the reason why you are conducting the survey. The following questions will help you to clarify these: • What are you trying to learn from the

survey results? • Who is the target population - who will

you be surveying and why? • Who is your audience - who will use

the information from your survey? • How will the information be used?

2 Write well constructed questions based on the objectives The questions you include in your survey should always be guided by your objectives. This will help to ensure that you gather quality data and can address both your needs and

YES

NO

TRAINING & DEVELOPMENT | FEBRUARY 2015 | 9WWW.AITD.COM.AU

those of your audience. Asking a set of standard questions is one thing, but to truly evaluate participants’ reactions to any program, their responses must eventually be collated and analysed. This may include identifying themes or patterns in qualitative responses, calculating means, medians and averages, categorising open-ended questions, comparing and contrasting different states, facilitators or courses, looking at changes in satisfaction levels over time and more.

Gathering quality data is dependent upon the quality of the questions that you have constructed. Ensure your questions have the following characteristics: • Clear and unambiguous – Use simple language. Not all

respondents will be familiar with complex terminology.

– Be specific. Your questions should be precise enough that the respondent is able to identify what the question is referring to without being overly wordy.

– Avoid double-barrelled questions. Double-barrelled questions include multiple parts, but ask for a single answer. Focus on one question at a time.

– Avoid double negatives. Respondents can become confused when reading questions and responses that both contain the use of negative words, such as “not,” “no,” or “didn’t.”

• Be concise. Respect your respondents’ time. Any survey which takes more than 15 – 20 minutes to complete at the end of a course should be reviewed for its relevance to the research objectives.

• Free of bias or leading statements. If questions are biased or leading in any way, they will steer a respondent toward the response that is considered socially desirable. Questions should be worded neutrally.

• Avoid or minimise sensitive topics. Asking respondents about sensitive topics can make respondents uncomfortable or embarrassed and should be avoided.

• Respect respondent privacy: – From an ethical standpoint this is very

important. Your survey respondents trust you with their information. It’s important to respect that trust and keep their information confidential.

– Anonymity should be protected, so make sure you avoid the common practice of asking participants to hand their completed forms directly to the facilitator.

– Responses should be de-identified in any reporting of results. If you wish to collect demographic information, collect it at the end of the survey, and allow respondents an opt-out if they don’t want to provide it.

3 Determine the format of the response options There are many different types of survey questions and each is suited to different goals. Chances are, one type of question will not suit all of your needs, so don’t be afraid to mix it up – this keeps it interesting for respondents as well as your stakeholders when it comes to reporting. Examples of question types include: • Yes/No options. These options are

quick and easy to answer and allow you to generate simple comparisons.

• Multiple-choice options. These options provide a fixed set of answers to choose from and can be designed to allow the respondent to select only one or multiple response options.

• Likert scales. These response options are used when measuring opinion- based questions. Respondents are asked to rate their preferences, attitudes, or subjective feelings on a scale.

• Open-ended responses. Open-ended responses allow the respondent to provide their own free-form answer. For large numbers of participants, open-

ended responses can be very time consuming to analyse, however they can be a very rich source of qualitative data. Use them sparingly, and only ask questions if you are prepared to analyse and report on the responses.

• Alternative responses. Several alternative response options are commonly used in surveys to allow the respondent to essentially “opt-out” of answering or provide their own answer to a question.

• Ordinal/Ranking. If you have a series of items that you would like respondents to rank, then a ranking question should suit your needs. For example: “Please rank each of the following items from 1 to 5, where “1” is most important and “5” is least important. Please use each number only once.”

4 Format the survey The format of your survey can have a great impact on your response rate. Poorly organised surveys may confuse participants, who may then not complete the entire questionnaire or refuse to participate. • Begin with an introduction. This

is your opportunity to explain the purpose and convince respondents that participating is worth their time and effort.

• Logically order and group the questions. Group questions by topic and place these in a logical order. For example, you might ask questions about the course logistics and administration, followed by curriculum pitch and content, followed by a section about the facilitator. Your initial questions are critical to ensuring continued participation so should be impersonal and easy to answer.

• Keep it short. Shorter surveys are more

Gathering quality data is dependent upon the quality of the questions that you have constructed.

10 | FEBRUARY 2015 | TRAINING & DEVELOPMENT WWW.AITD.COM.AU

likely to be completed by respondents. The ideal survey is short while still capturing all of the information necessary to meet its objectives.

• Use contingency questions. If relevant, use contingency questions which prompt the respondent with a preliminary question to determine if further questions will apply. Respondents will not have to weed through questions that do not apply to them.

• Use a progress bar or page progress indicator for online surveys. Indicate the number or percentage of questions remaining will give the respondent a sense of the length of the survey and the amount of time remaining.

• Close with a thank you. Always thank respondents for their participation. You may also wish to provide them with your contact details for any questions. If you will be sharing any reporting of results, offer them an opportunity to opt in if relevant or possible.

More on likert scales The most widely used rating scale is the Likert Scale. This is a multi-point scale, which is used to allow the individual to express how much they agree or disagree with a particular statement or series of statements about a topic.

Each of the responses has a numerical value, which can be used to measure the attitude under investigation. Likert Scales are useful as they allow for degrees of opinion, and even no opinion at all. Quantitative data is obtained and can easily be analysed.

Likert scale best practices and tips • Use labels. Numbered scales, eg scales

that are only marked from 1 to 5, can create confusion. Add words to label your scales, eg ‘poor’ (1) ranging to ‘excellent’ (5).

• Use an odd number. Scales with an odd number of values (eg 1-to-7, 1-to-5) will have a midpoint. To elicit balanced feedback, you need to ensure your feedback scale is balanced. Forcing choice by not offering a neutral option

can lead to lower overall scores. Studies have shown that more than 7 options are too many and 5 is generally ideal. There is no rule as to whether values should decrease or increase from left to right, but ensure you follow a consistent direction throughout your survey.

• Use equal spacing. Response options in a scale should be equally spaced from each other.

• Keep it inclusive. Scales should span the entire continuum of responses. For example, if a question asks how well prepared the facilitator was and the answers range from “extremely well prepared” to “moderately well prepared”, respondents who felt the facilitator wasn’t prepared at all won’t know which answer to choose.

References Czaja, R. & Blair, J. 2005, Designing surveys – a guide to decisions and procedures, Sage Publications, London. Denscombe, M. 2010. The good research guide for small-scale social research projects. Open University Press, New York, NY. Rea, L.M. & Parker, R.A. 2005. Designing and conducting survey research – a comprehensive guide. Jossey-Bass, San Francisco, CA. Sapsford, R. 2007, Survey Research. Sage Publications Ltd, London.

Lisa Elias is a Sydney-based learning and development specialist and the Director of OD Analytics and Sticky Learning. Lisa offers end-to-end L&D/OD services, focussing on needs analysis, capability development and the evaluation of effectiveness, impact and ROI. Contact via [email protected]

CHECKLIST FOR THE USE OF QUESTIONNAIRES Before implementing your questionnaire ensure you can confidently answer yes to the following questions:

1. Has the questionnaire been piloted?

2. Is the layout clear?

3. Is there an explanation of the purpose of the research?

4. Have thanks been expressed to the respondents?

5. Are there assurances about anonymity and the confidentiality of data?

6. Are there clear and implicit instructions on how the questions are to be completed?

7. Have the questions been checked to avoid any duplication?

8. Are the questions clear and unambiguous?

9. Have all the non-essential questions been excluded?

10. Are the questions in the right order?

11. Will closed questions produce the required kind of numerical data?

12. Have you included open text options sparingly and carefully?

13. Have you allowed sufficient time to complete it?

Copyright of Training & Development (1839-8561) is the property of Copyright Agency Limited and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.