Customer Satisfaction Survey

Survey construction

Author(s):

Joseph Janes (Joseph Janes is at the School of Library and Information Science at the University of Washington, and is founding director of the Internet Public Library. [email protected])

Abstract:

This column continues a series on topics in research methodology, statistics and data analysis techniques for the library and information sciences. It discusses surveys, how to write good survey questions, questionnaire design and construction, including the order of questions, instructions, design and layout, and gives suggested readings and references.

Keywords:

Libraries ,  Methodology ,  Questionnaires ,  Research ,  Statistics ,  Surveys

Type:

Research paper

Publisher:

MCB UP Ltd

Copyright:

© MCB UP Limited 1999 Published by MCB UP Ltd

Citation:

Joseph Janes , (1999) "Survey construction", Library Hi Tech, Vol. 17 Issue: 3, pp.321-325,  https://doi-org.ezproxy.apollolibrary.com/10.1108/07378839910289376

Downloads:

The fulltext of this document has been downloaded 6155 times since 2013

Article

Surveys. We have all seen them, in many facets of our lives, personal and professional. Many of us have even designed and sent out surveys, and the purpose of them is almost always to find out what is going on out there, what people are thinking or doing, and perhaps how things are changing. Unfortunately, many survey instruments (this is the proper term – most people just call them questionnaires) are poorly designed with badly worded questions and ill‐conceived analyses of the data that come back. In such cases, it is really impossible to know to what extent the data are a reasonable reflection of the phenomena or people being surveyed, and thus are a massive waste of time for just about everybody involved.

In this column, I shall try to help you through some of the important ideas and practices in constructing a good survey.

Surveys

Section:

Choose

A survey is a good way, often the only way, of getting a picture of the current state of a group: a community, an organization, an electorate, a set of corporations, a profession. We might think of surveying doctors about their opinions about a new therapy, voters about their preference for President in the next election, librarians about how often they use the Web in reference interactions. In many cases, these surveys are snapshots, pictures of a particular point or period in time, although there are longitudinal surveys which take place over longer periods.

There are several major steps in building a good survey. In this column, we shall talk about two: writing good questions and designing good questionnaires.

These steps do not happen in a vacuum, though. Before beginning the hard work of developing a survey and asking for people’s valuable time in responding, ask yourself some hard questions. Is this survey worth it? Is what I will learn from this valuable enough to justify my time in building it and other people’s time in answering? Can I carry it out well? Do I have the necessary skills and resources to make it successful? What do I want to be able to do with the results when it is over?

If, after all of that soul‐searching, you still want to proceed, the whole process goes something like this:

•

get an idea;

•

see if anybody else has done a similar survey;

•

decide what you want to know;

•

decide on your population of interest (citizens, librarians, high school students);

•

write a bunch of possible questions;

•

design a questionnaire;

•

pretest the questionnaire;

•

modify the questionnaire based on the pretest;

•

draw a good sample from your population to survey;

•

administer the questionnaire;

•

analyze the data;

•

draw conclusions.

Sharp‐eyed readers will notice that this list bears a strong resemblance to the list of steps in the research process from the first column in this series:

•

identify the problem;

•

read literature;

•

identify methodologies;

•

collect data, analyze, evaluate;

•

draw conclusions.

with a few more elaborate steps. Conducting a good survey which produces results you can trust is difficult work. If you are serious, I suggest getting hold of the resources I list at the end for further and more detailed help.

Writing good questions

Section:

Choose

This sounds simple enough, but believe me, the art of writing a good, nonbiased, answerable question which will give you good data and insight into your problem is tricky indeed. The most important thing to remember is that people will answer the question you ask them, not necessarily the question you wanted to ask them. The words you use and the way in which a question is constructed can have profound implications for the answers you get.

What follows is a checklist drawn from several sources, including the works of Babbie (1990), and personal experience. It is not an exhaustive list, but it is a good set of guidelines of things to think about when writing questions. Good questions are:

•

Related to the problem at hand. Since people will answer pretty much anything you ask them (within limits), do not ask anything you do not really need to know. If your survey is about the way pet owners feel about and treat their pets, it is probably not relevant to ask about the kinds of cars they drive or their annual income.

•

The correct type to get the best information. Options include:

· – multiple‐choice (if there are good, logical choices);

· – open‐ended (if there are many different possible answers, though this can be harder to analyze), and

· – yes/no/maybe/don’t know, strongly agree/agree/no opinion/disagree/strongly disagree.

•

Clear, unambiguous, precise. Give definitions where appropriate; avoid jargon or slang unless needed or appropriate (e.g. surveying teenagers, professionals).

•

Not leading. “Don’t you think that…” is almost always a bad way to start a question. Let them tell you what they think.

•

Able to be answered by the subjects. Do not ask questions they cannot answer or do not know. It irritates and embarrasses people, and they will make up answers or refuse. In particular, do not ask for too much detail. It is impossible for people to tell you how many library books they borrowed in the last year; give them ranges instead and have them pick one, or ask for how many they borrowed on their most recent library visit.

•

Not double‐barreled. The word “and” is a sign; it often indicates that you are asking two questions in one, and that is usually not good. “Do you think we should increase funding for libraries and use that money for buying more books?” is two separate questions and should be asked that way.

•

Short. People will not read too much, get confused easily, and have short attention spans (a problem with phone surveys).

•

Not negative. Avoid the word “not” in a question; it can easily be misheard or not heard and thus changes the question. “How many times in the last week have you not driven to work?” is bad; try “How many times in the last week did you take public transportation or car pool to work?” instead. Of course, this is a slightly different question, but the phrasing is much better.

•

Unbiased. Some surveys have a point of view in mind; they are trying to systematically influence the answers. “Don’t you agree that in order to protect our family values and American way of life, the Second Amendment should be repealed to take guns out of the hands of irresponsible people?” is an unusual but by no means extreme example. For controversial topics, take special care to be as neutral as possible; small changes in wording can make a big difference.

Below are a couple of examples of questions (and the answers they got) for you to think about. Both of these come from the POLL database (Public Opinion Online) on DIALOG (File 468) (see  Appendix ):

1.

(1) Do you think that books that could show terrorists how to build bombs should be … banned from public libraries, or available in the library like every other book?:

•

Banned from public libraries – 70 percent

•

Available in the library like every other book – 25 percent

•

Don’t know/Refused – 5 percent.

1.

These extreme results came from a national sample of 1,026 adults, asked over the telephone, in a survey conducted in summer of 1997 for the Freedom Forum by the Center for Survey Research and Analysis. The question is pretty good, but you can see how a controversial subject can produce striking results. (According to their Web site, The Freedom Forum is a “nonpartisan, international foundation dedicated to free press, free speech and free spirit for all people”.)

2.

(2) (Do you agree or disagree with each of the following statements?)

3.

We must connect every classroom and library to the Internet by the year 2000 so that all children have access to the best sources of information in the world:

•

Strongly agree – 44 percent

•

Somewhat agree – 21 percent

•

Somewhat disagree – 16 percent

•

Strongly disagree – 14 percent

•

Neither – 1 percent

•

Don’t know – 4 percent

1.

These results came from a national sample of 1,000 registered voters in February 1997 who said they would definitely or probably vote in the 1998 election. The survey was conducted by American Viewpoint, a polling firm whose Web site lists many corporate and political clients (all Republicans, as far as I can tell). This question is what I call a “fluffy bunny” type; it is pretty difficult to say no to giving children “access to the best sources of information in the world”. One wonders what the numbers would have been if it had been worded more neutrally.

Questionnaire design and construction

Section:

Choose

Once you have got the questions written, you need to put together the questionnaire or instrument you are going to use to administer the questions to your subjects. The design you use will depend on several factors, chief of which is the method you are going to use in asking the questions: telephone, face‐to‐face, or self‐administered (mail, handout). There are aspects to consider within each of these; the guidelines below are fairly generic, though some will pertain to only one type.

Order of questions

Section:

Choose

The order in which you ask questions may have an impact on the answers you get. If you ask a question in a sensitive or personal area, some traces of the question may well linger on; asking about a traumatic event, for example, might bring up memories of that event that will not easily or quickly go away. We could simply randomize the order of the questions to diminish this effect, but a good survey should flow to make it easier for subjects to answer, so we need to be sensitive to order issues. Pretests usually help to catch problems of poor ordering, especially in an interview where you can pick up on body language, nonverbal cues, and so on. (Indeed, it often is a good idea to do a small face‐to‐face pretest of a survey you intend to mail to people just to look for these kinds of reactions.) There is some evidence that if you survey people with higher levels of formal education, the effect of question ordering is lower.

For self‐administered questionnaires, it often helps to lead off with the most interesting, non‐threatening questions to get people to start answering questions, leading to the tricky or problematic questions toward the end, and finish off with demographic questions (age, gender, educational level, unless of course these are the tricky ones) which are quick and easy.

For face‐to‐face surveys, it helps sometimes to start with the demographic questions to establish a rapport, and then flow to the interesting ones.

In any case, you want people to finish the survey. If you are interviewing them, and can establish that rapport, they are less likely to stop or walk away. In a mail or handout survey, they can stop at any stage and once they do, you may well have lost them. Make it as easy, fun, interesting and worthwhile to them as you can.

Instructions

Section:

Choose

Be specific. Remember that people will answer the question you ask, but will not necessarily do what you tell them. Start with general information; the purpose of the survey, who you are, if you represent an agency or organization (unless revealing that would flip them out, in which case you may conceal it but you will have to have a good reason for that). Tell them what you want them to do (make an X, circle), how long the survey is likely to take, how long their answers should be, how to make additional comments, and so on.

Then, for each section, provide a short introduction about its content and purpose (“This section will ask about services your library provides to children.”). This provides a framework for them to think about the questions and for their answers.

If particular questions require specific instructions, give the instructions at that point. Examples include questions with multiple parts, or questions that require a different type of answer (e.g. Here, choose the one best answer).

In a face‐to‐face or telephone interview setting, it is best to provide your interviewers with a script that they can read verbatim and to train them in how to conduct the interviews. This will somewhat reduce random errors that can creep into responses due to different interviewer styles, voices, and so on. The exception is for semi‐structured or unstructured interviews where there are few if any scripted questions and the point is just to get the person to talk and see what happens.

Design and layout

Section:

Choose

These guidelines are specific to self‐administered questionnaires. Try to maximize white space in the form, make it as uncluttered and clean as you can, but do not make the survey too long (the longer it is, the less likely people are to complete it). Do not abbreviate. Leave enough space for answers (and realize that people will use the amount of space you give them as a guide to how much you want them to say). Make it attractive. Make options and directions clear (“If YES, then go directly to Question 15; if NO, go on to Question 11”, with big black arrows if possible). If you tell people to check a box or circle something, make sure it can be done without confusion.

The resources I list below have lots of examples of good and bad design to help you with all of this. Much of it is common sense, but it is both an art and a science to design a good survey to get quality data that will help you to know what is going on in your area of interest. A bad survey is nothing more than a waste of time, but a good, well‐designed, well‐analyzed survey can be an invaluable source of data.

In the next column, we shall talk about analysis; some simple techniques for displaying, summarizing and understanding the data you get from surveys and other investigations.

Reference and further reading

1.

Babbie, E. (1990), Survey Research Methods, 2nd ed., Wadsworth Publishing, Belmont, CA.I have been a big Babbie fan for many years, and this is the book I turn to when I need help. Written with a humorous tone and with great examples, it is an excellent place to begin, and includes useful context on the research process as well. [Google Scholar]

2.

Fink, A. and Kosecoff, J. (1998), How to Conduct Surveys: A Step‐by‐Step Guide, 2nd ed., Sage Publications, Thousand Oaks, CA. A very practical and useful guide to survey methodologies, with lots of interesting examples (largely but not exclusively from the health sciences), well written and quite comprehensive given that it is only about 100 pages long. A handy tool for reference. [Google Scholar]