Business Research

1

Questionnaire Design The use of a formal questionnaire helps ensure the comparability of the data collected across both interviewers and respondents and, as a result, permits statistical analysis of this data. Properly designed questionnaires increase the speed with which the data can be collected and promote greater accuracy in the recording and handling of the data. For example, good questionnaire design can contribute to the ease of administration, editing, and coding of the questionnaire. More importantly, good questionnaire design is critical to the validity of the data collected. For large scale surveys (that is, sample sizes of 500 or more), the error due to poorly worded questions is likely to be much greater than that of the random sampling error accounted for in confidence intervals. Novices at questionnaire design – and too frequently those with a moderate amount of experience – tend to falsely assume that if subjects complete the questionnaire their responses are meaningful. Unfortunately, respondents will typically select a response to the most meaningless of multiple choice questions without any hesitation and never say anything to the interviewer about the fact that the question made no sense to them. For example, one study in the late 1950s asked respondents the following question: Do you approve or disapprove of the Metallic Metals Act of 1956? Approve ______ Disapprove ______ No Opinion ______ Seventy percent of the sample expressed an opinion, either approval or disapproval of this legislation even though it never existed. Questionnaire Design Process Following is an outline of the questionnaire design process. Each of these steps will be dealt with in detail in the following sections: 1) Specify information needed 2) Specify type of interviewing method 3) Determine question content 4) Choose question structure 5) Choose question wording 6) Determine question sequence (order) 7) Design form/layout/code for data entry 8) PRETEST-PRETEST-PRETEST!!! Specify Information Needed The first step in questionnaire design is to specify the information needed. Note that this was also the first and most important step in the research design process – as the research project progresses, the information needed becomes more and more clearly defined. It is helpful at this stage to review the components of the managerial and research problem, the hypotheses, sample, research design,

2

etc., to help focus your thinking as you prepare to create a questionnaire. The researcher must then sit down and outline the exact information that s/he wants to collect from this survey instrument.

Just as important at this stage as deciding what information will be collected are the decisions about how that information will be analyzed. The researcher should create a “dummy table” at this stage – a dummy table is a blank table used to catalog data as it will be entered into the analysis software. It describes how the data will be structured once the data have been collected and will help the researcher determine how the information can be analyzed. Without an analysis plan in place PRIOR to creating and administering the questionnaire, it is likely the data will be collected in forms that prove to be unusable in testing the required hypotheses.

Specify Type of Interviewing Method The type of interviewing method – mail, telephone, personal or internet – was addressed in the reading Chapter 10. Review that chapter for help in deciding on the type of method to use for a particular information need.

Determine Question Content The content of each question to be included in the questionnaire should be guided by the specification of information needs in the problem definition stage, i.e., your investigative questions. Obviously, if the specification of information needs is not sufficiently precise, it will be difficult to determine which questions are necessary and what their exact content should be. One of the objectives of good questionnaire design is to write the questions to overcome respondents’ unwillingness to answer and their inability to answer accurately. The following discussions of question content, structure and wording all have an impact on respondents’ ability and willingness to respond accurately. There are several factors one should consider in designing the content of the questionnaire:

Is a particular question that has been proposed for inclusion in the questionnaire really necessary? Managers frequently want to include questions they feel are "interesting" but don't know exactly how they will analyze, or use them once the data are collected. Somehow they imagine that once they see the results of these questions they will know what to do with them. Unfortunately, it rarely happens that the managerial relevance of including a particular question becomes clearer after the data are collected. Instead, the reverse is likely to be the case because in most studies the results usually do not strongly favor one alternative course of action over another.

There tends to be an incorrect perception that a few extra unnecessary questions will not increase the cost of the study very much and that one doesn't need to worry about how all of the questions are going to be analyzed before the survey is conducted. Although it is true that adding a few extra questions to the questionnaire may not increase the field costs (printing, mailing, and interviewer costs) very much, there are other "costs" to be considered. The amount of time and effort a respondent will devote to a particular survey does not increase proportionately with the length of the questionnaire. For example, if respondents would spend an average of 5 minutes with a one-page questionnaire, they are likely to spend perhaps only 8 minutes with a two page questionnaire on the same general topic. This suggests that respondents will give less thought to each question and, therefore, provide less accurate data.

3

Also, the increased size of the questionnaire is likely to reduce the response rate, particularly for mail surveys, and thereby increase the cost per completed interview and raise the level of non- response bias. Finally, including unnecessary questions is likely to create confusion in analyzing the results and increase the length of the final report. For example, with 10 questions there are C(10,2) = 45 possible two-way cross classification tables one could construct compared with C(20,2) = 190 for a 20 item questionnaire. Does the question relate to the precise issue of interest? Not infrequently, the question as worded does not relate to the exact issue of interest. For example, if you were to ask respondents where they most frequently shopped for clothes, they might specify fairly high priced specialty stores. They may shop there to find out what is in style and to look for sale merchandise. However, they may buy most of their clothes at stores like JC Penney's because that store’s clothes are more affordable.

Is more than one question required? Sometimes it is necessary to use a "screening question" to determine if a particular questionnaire item is meaningful for the respondents to answer. For example, one should not ask respondents what brand of computer they own without first determining that they own one. Occasionally, researchers will formulate a question that contains multiple issues. For example, consider the following item:

Do you think Tide gets clothes clean without injuring the fabric? Yes ______ No ______

If consumers answered "No" to this item, there would be no way of determining whether it was the "clean" issue, the "injury" issue, or both that prompted them to do so.

Can the respondent be expected to answer the question accurately?

Questions that may seem unreasonable to the respondent should be avoided, or, if /they are important, the respondent should be provided with an explanation for their inclusion. Most respondents don't understand why interviewers inquire about such things as their education or income in a survey that was presented as being focusing on some product or service. Furthermore, many will refuse to respond to items of this type unless they are provided with a short but clear explanation of the interviewer's need to collect this data (e.g., “I would like to ask you a few questions about yourself to help us describe the participants in our sample”). When these questions are asked they should come last in the questionnaire and this data should only be collected at the level of detail necessary for the study. For example, if in the data analysis income is to be divided into three groups (e.g., less than $10,000/year, between $10,000 and $29,999/year, and $30,000/year or more), these groupings should be used as response categories in the questionnaire rather than asking respondents for more specific (and thus more personal) descriptions of their income.

4

Determine Question Structure The structure of questions can have a great deal of influence on whether subjects are able and willing to answer accurately. Questions can be either unstructured or structured. Unstructured questions are those that are open- ended, in which respondents are expected to fill in a blank by answering in their own words. Examples of unstructured questions include: What is your occupation? What do you think of people who patronize discount clothing stores? Who is your favorite political figure? Unstructured questions should be very limited in survey research. By the stage in the research process in which descriptive research is appropriate, the researcher should be past the exploratory phase and should have very well defined information needs that can be collected in structured question formats. However, unstructured questions might be useful as first questions on a topic in a personal or telephone interview (however, never include them as a first question on a written survey – they require too much “work” on the part of respondents and will greatly increase non-response rates). As an opening question during an interview, unstructured questions allow respondents to express general attitudes and opinions that can help researchers interpret responses to structured questions. However, the potential for interviewer or interpreter bias is high on unstructured questions, just as it is for exploratory interviewing techniques. Coding of responses is time- consuming and costly. Unstructured questions also give extra weight to the opinions of respondents who are more articulate or more motivated to respond. Most importantly, recognize that unstructured questions are rarely appropriate for written survey techniques – either mail questionnaires or email/internet surveys. Even if subjects do respond to them, people tend to be much more brief in writing than in speaking. More importantly, self-administered (written) surveys have much higher refusal rates for participation given the additional effort perceived in answering unstructured questions. If you find yourself designing a questionnaire that requires a large number of unstructured questions, you probably ought to back up and do more exploratory research before attempting a survey. In contrast, structured questions provide some sort of organized response categories. Structured question response categories can be dichotomous, multiple-choice or scales. A dichotomous question is one with two possible answers, for example:

Are you the primary purchaser of groceries in your household?

____ Yes ____ No Some examples of typical multiple-choice questions include:

What was your annual household income in 2000, from all sources, before taxes? ____ Less than $20,000

5

____ $20,000 to $39,999 ____ $40,000 to $59,999 ____ $60,000 to $79,999 ____ $80,000 to $99.999 ____ $100,000 or more

Which of the following grocery stores have you purchased products in within the last month? (Check all that apply) ____ AJ’s Fine Foods ____ Basha’s ____ Fred Meyers ____ Fry’s ____ Safeway ____ Whole Foods Market ____ Other (Please list here: ________________________) Do you intend to buy a car within the next six months? ____ Definitely will not buy ____ Probably will not buy ____ Undecided ____ Probably will buy ____ Definitely will buy

Scales have been dealt with in detail in the “Measurement and Scaling” portion of this course, and are described in the “Scaling Techniques” reading section. Review those readings again if necessary. Therefore, following is a more detailed discussion of issues regarding the design of dichotomous or multiple-choice structured questions. When using multiple choice questions it is important that the response categories be mutually exclusive and collectively exhaustive. The mutually exclusive requirement specifies that there can be no overlap in meaning between any of the response categories. The collectively exhaustive criteria stipulates that all meaningful responses be listed. For example, consider the following question.

How do you most frequently get to work? ____ personal car ____ motorcycle ____ car pool

____ public transportation ____ walk

____ bus These responses are not mutually exclusive because a bus is a form of public transportation. Some respondents who most frequently use the bus to get to work will check "public transportation" while others will check "bus" – the analyst will have no way of telling how many of those who checked "public transportation" took the bus and how many used other forms of public

6

transportation. Furthermore, some respondents may most frequently ride a bicycle, a response that is not listed. Therefore, the response categories also are not collectively exhaustive. One can use an "other (please specify)" category to overcome the problem of not having a collectively exhaustive set of response categories. However, past research has shown that respondents are reluctant to check this category. That is, there is a strong bias against those alternatives that are not specified in the list. If in the above example, an "other (please specify)" category were added, only about half of those respondents who rode bicycles to work would be likely to check this category and write in that they rode bicycles to work. In some sense it is quite like being a "write-in" candidate in an election. Another issue relating to dichotomous or multiple choice questions is the order in which the responses are listed. There is a strong bias in favor of the first alternative listed when the responses are verbal (i.e., qualitative) and the issue of interest is attitudinal. There also exists a lesser bias in favor of the last alternative under these same conditions. For example, candidates in an election in which there were several people running for the same office should most prefer to be listed first on the ballot. Their second choice would be to be listed last. One way to avoid order bias for these types of questions is to rotate the response categories as one conducts the survey (that is, each alternative is listed an equal number of times in each position). When, instead, the question of interest asks for some type of numerical estimate, order bias has the opposite effect. Subjects will tend to select a response from the middle of the rank ordered numerical categories (that is, there is a bias in favor of the middle alternatives). This bias exists because respondents often assume that the more extreme numerical values are less likely to occur. This problem can be partially overcome by including a wide range of numerical categories (i.e., the end categories should sufficiently extreme that few respondents would be likely to give answers that fall in these categories if they were responding to an open-ended question rather than a multiple choice question). Determine Question Wording There are several problems with question wording that frequently occur other than those associated with using ambiguous words, unfamiliar words, and/or response categories that are either not mutually exclusive or not collectively exhaustive. Many these are discussed in a classic book by Payne called "The Art of Asking Questions." Several of these problems are illustrated and discussed below.

Implied alternative questions A implied alternative question is one in which one of the alternatives is not stated in the

question, but rather only implied by the question. This type of problem most frequently occurs with dichotomous questions (questions for which a "yes" or a "no" answer is expected). For example, consider the following question:

Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year? Note that the alternative that "layoffs are avoidable" (which would correspond to a "yes" response) is stated in the question while the other alternative (i.e., "layoffs are not avoidable," which would correspond to a "no" response) is not stated in the question but rather is only implied by the question. When this particular question was asked in a survey, 63% of respondents answered "yes,"

7

indicating that they felt "layoffs are avoidable." To "correct" an implied alternative question one need only state all alternatives within the question. For example, the above implied alternative question might be corrected by adding the phrase "or do you feel layoffs are unavoidable" as in the revised question below: Do you think most manufacturing companies that lay off workers during slack periods could arrange things to avoid layoffs and give steady work right through the year or do you feel layoffs are unavoidable. When this revised question was asked in different, but comparable survey, the results were dramatically different. Only 35% of the respondents in this second survey replied that they thought that layoffs were avoidable, a difference of 28% (i.e., 63% - 35% = 28%). Obviously, implied alternative questions can create a strong bias in favor of the stated alternative when the issue of interest involves attitudes or subjective judgments, particularly if a substantial proportion of those sampled does not hold strong opinions. This is because many people will merely agree with whatever position is stated when they don't feel strongly about an issue. However, implied alternative questions are not typically a problem when the issue is factual in nature (for example: “Do you own a pet?”).

Wide open or not? "Wide open" questions are questions which are insufficiently precise for reasons other than using words that are considered to be ambiguous. They fail to precisely define the issue of interest because they refer to a category that is much too broad. For example, consider the following item:

What brand of soap do you like best? The problem with this question is that it is not clear whether it is soap for bathing, washing dishes, washing clothes, or some other type of soap that is of interest. One might correct this question by merely specifying the type of soap that is relevant as was done in the revised question below.

What brand of dishwashing soap do you like best?

Complementary alternatives As a general rule, when dichotomous questions are used to measure respondents' opinions on some issue, the alternatives presented should be equally extreme. Below is a question that violates this principle. Would you say it's better to regulate business pretty closely, or would you say the less regulation of business the better? The alternative "the less regulation of business the better" suggests that it is best to have no regulation of business at all. As such, this alternative is more extreme than the other alternative (i.e., "it's better to regulate business pretty closely"). Since most people tend to avoid taking extreme

8

positions, there will be a bias in favor of the less extreme alternative. To correct the problem one needs only to replace the more extreme alternative with a more moderate alternative. Pie a la mode A "pie a la mode" question is one that provides alternatives two different but related issues. For example, consider the question below: Would you say that the mayor is doing a good job of running the city or that he could do a better job? At first glance this question appears to be quite reasonable. However, it does involve two related issues: 1) is the mayor doing a good job or not; and 2) could the mayor do a better job or not. It is possible that a respondent might feel that the mayor is doing a good job but still think that he could do a better job than he is currently doing. (Many of us have heard our parents say to us that we did well in school but that we could still do better. We have also been asked whether we wanted ice cream or pie for dessert when we wanted both - i.e., pie a la mode) Hence, the respondent might agree with both alternatives. To correct "pie a la mode" questions one needs to formulate two new questions, one for each of the issues represented in the original question as illustrated below. Would you say that the mayor is doing a good or a poor job of running the city? Do you think the mayor could do a better job of running the city or that he is not capable of doing better? Note: When correcting a pie a la mode question, it is important that you not create two implied alternative questions.

Balance Some questions are "unbalanced" in their alternatives in the sense that some "ideas" are represented more than once in the alternatives. For example, consider the following question:

Which one of these things most influenced your decision when you most recently purchased a hat?

____ style ____ good looks ____ workmanship ____ material ____ appearance

The problem with this question is that "how the hat looks" is represented three times in the set of alternatives (i.e., style, good looks, and appearance all relate to how the hat looks and mean much the same thing). When an idea is represented multiple times in a set of alternatives it tends to be chosen more often. Therefore, to correct a question of this type one need only delete all but one reference to any given concept. In the case of the above question, one might want to delete "good looks" and "appearance" and retain "style."

9

Double-barreled questions Double-barreled questions involve alternatives which are combinations of positions to two different issues. Furthermore, the list of alternatives often does not include all possible combinations. For example, consider the following item:

Do you prefer to date people who are short with dark hair, or tall with blonde hair? Obviously, some people might prefer short and blonde, or tall and dark, the two missing alternatives. Hence, the respondent may not want to select either of the two alternatives. One could either list these two missing alternatives or, preferably, create two separate questions (assuming, of course, that the issues are independent). For example, the above double-barreled question might be corrected by splitting it into two questions in the following manner.

Do you prefer to date people who are short or tall? Do you prefer to date people who have dark or blonde hair?

How specific?

These questions require the respondent to estimate a percentage but fail to specify the base on which the percentage is to be calculated even though many bases are possible. For example, consider the following item.

About what percent of after tax profit would you say that your company made last year? There are many different bases which one could use to calculate a profit percentage (for example: gross sales, net sales, stockholders' equity, assets, etc.). These different bases would provide very different percentages for a given level of profit measured in dollars. One might correct this problem as follows: About how much after tax profit as a percent of sales would you say that your firm made last year?

Extensive questions

Extensive questions are questions that require respondents to engage in a considerable amount of mental calculations to arrive at a reasonably accurate answer. For example, consider the following item.

How many eggs do you fry in a year? Many people will not have a good idea of how many eggs they fry in a given year stored in their memories. It is something to which most people give little, if any, thought. As such, they must develop an estimate. They could try to recall individual instances of frying eggs and add these together. However, this would obviously take a very long time and people would not recall many of these instances. Probably the best way to develop such an estimate would be to think of how many eggs they fry in a typical week and multiply this figure by 52. However, not everyone will think of this method. A good way to overcome this problem would be to shorten the time period over which the estimate is to be made as in the revised question below.

10

About how many eggs do you fry in a typical week?

If the researcher wants an estimate for an entire year, he/she can multiply the respondent's reply by 52. This could be done by computer and would be much more accurate than asking the respondents to make these calculations.

Uncomplimentary exceptions These questions involve exceptions to the general rule that the alternatives to dichotomous questions should be equally extreme. For example, consider the following items:

Do you think salaries of teachers like yourself are too low or too high? Few individuals are likely to think they are overpaid and even less likely to admit that they might be overpaid even if they felt that way. Hence, given the above question, almost the entire sample would be "forced" to say that their salaries were too low even though many might think their salaries were about right. To correct this problem one could change the "too high" alternative to "about right" as indicated in the revised question below.

Do you think salaries of teachers like yourself are too low or about right? Determine question sequence Information types and order of collection There are three types of information that can be collected on a questionnaire: 1) basic information; 2) classification information; and 3) identification information. Basic information is the critical information that relates directly to the research project – the questions that answer the key information needs of the survey (for example, capturing respondents opinions and attitudes about the product or service of interest.) Classification information is the socioeconomic, demographic and lifestyle characteristics (for example, age, income, gender, marital status, occupation, number of children in household, etc.) of respondents that are used to classify respondents and interpret their answers. Identification information includes respondents’ names, addresses, telephone numbers. Identification information does not have to be collected on a survey – in fact, it rarely is because researchers usually want to offer respondents the opportunity to answer freely by remaining anonymous. Identification information may be collected for a few legitimate reasons. For example, identification information may be needed if field service supervisors need to verify that respondents were actually interviewed, or to remit promised incentives to respondents for participating, or if follow-up contacts are needed to collect additional data and link it to the information previously given by a specific respondent (for example, to measure changes in brand attitudes over time.) As a general guideline, basic information should always be obtained first in a questionnaire, followed by classification information, and finally, identification information if it is necessary. The basic information is of the greatest importance to the research study and should be obtained first, before we risk alienating respondents by asking a series of personal

11

(classification information) questions. Classification information is often boring to respondents. In addition, classification questions often pertain to information that respondents may consider private, so they may be sensitive about answering them before they have become involved and engaged in the questionnaire. As a result, if these questions come first, many respondents may decide that they don't want to participate in the survey and terminate the interview or throw away the questionnaire. When the classification questions are placed at the end of the questionnaire, the respondent will have already made an investment in the survey by answering the "product" oriented questions and will usually complete the interview or questionnaire. Therefore, never ask for demographic information at the beginning of a questionnaire.

Opening Questions The opening question, and in some cases the opening set of questions, should be designed to win the respondents' interest in the survey. The question should be easy to answer and should relate to something about which the respondents would like to express their opinions or perhaps would enjoy relating their experiences. If telephone or personal interviews are being conducted, an open-ended question will often get the respondent talking and involved in the survey. For example, if you were conducting a personal interview survey for a restaurant, you might want to open your interview with one of the following questions. How do you feel about the availability of good restaurants on Mill Ave.? Have you recently had any particularly enjoyable experiences at restaurants on Mill Ave.? In contrast, a written questionnaire should never open with an unstructured question. However the structured opening question should still meet the criteria of being interesting and easy to answer.

Difficult questions Place the difficult questions in the middle of the questionnaire so that the respondent will have had some time to think about the general topic area before being challenged with these items and will be able to give more thoughtful answers. Furthermore, if difficult questions are asked early in the questionnaire, the respondent may feel the study is too difficult for them and will terminate the interview.

Funneling The sequencing of questions within sections of the questionnaire is often important because earlier questions can sometimes bias the responses to questions that follow them. Consider the bias that is likely to occur given the order of the following two questions:

Do you ever watch “Survivor”? What are your favorite television programs?

The second question is likely to elicit many more references to Survivor when the questions are asked in this sequence than if their order were reversed. Not only does the first question make the mentioned program more salient to the respondent, but it also suggests that the sponsors of the study have a special interest in this program. However, asking about the subjects' favorite television

12

programs first is not likely to lead the respondent. If the specific program names must be addressed in a survey, they should appear in the following order. What are your favorite television programs? Do you ever watch “Survivor”? This procedure is referred to as “funneling” – that is, starting broad and getting more specific as the questionnaire continues, just as the image of the funnel suggests. Funneling is particularly necessary when brand perceptions are being measured. Therefore, as a general rule, don't ask questions that identify the sponsor of the survey until it is absolutely necessary to do so. Arrange the questions in a logical order (for example, how they first became aware of the store, what prompted them to shop there the first time, what they like and dislike about the store) and group the questions by topic (for example, how various brands rate on different product attributes). This will make it easier for the respondents to complete the questionnaire and they will be likely to provide more accurate answers. Layout and reproduction of the questionnaire The layout and reproduction of the questionnaire is particularly important for mail surveys for at least three reasons. First, these factors can have a strong effect on the response rates obtained in mail surveys. Two major considerations in many prospective respondents’ decisions of whether to participate in a mail survey are their perceptions of how much effort will be required to complete the questionnaire and their assessments of how important the study appears to be. Poorly designed questionnaire layouts and inferior reproduction methods will make the questionnaire appear more complex to complete and will create doubts about the importance of the survey. Most respondents will tend to believe that if the study were really important, the sponsors would have spent the time, effort, and money necessary to design a professional looking questionnaire. Second, if the layout or reproduction of the questionnaire is inadequate in some respect, it is likely that many of the respondents will make errors in recording their answers. For example, some respondents may not be able to follow the skip patterns in the questionnaire and respond to the wrong set of questions. Others may check the wrong response categories when these categories are placed close together. However, online survey software (e.g., Qualtrics) has eliminated this problem at large (e.g., you can set up skip logic using Qualtrics). Finally, a poorly constructed questionnaire can cause difficulties in coding and entering the data. For example, when the questionnaire is not precoded (that is, numerical values are not pre-assigned to the response categories and printed on the questionnaire), this function will have to be performed manually at greater expense and with a higher error rate. Furthermore, when response categories are placed close together, it is likely that those entering the data will enter the wrong values. Again, online survey software (e.g., Qualtrics) has elimated this problem at large (e.g., you no longer enter data manually as Qualtrics records the data automatically). Increased errors in recording the raw data and entering it into the computer for data analysis can also occur in telephone and personal interviews. Furthermore, inadequate layout and reproduction of the questionnaires can increase interviewing time, and, hence, the cost of the survey.

13

There are some generally accepted rules that one should keep in mind when designing the layout and deciding on the reproduction of a questionnaire:

Precode all multiple choice and dichotomous questions.

Avoid making the questionnaire pages appear crowded. There should be a lot of "white space" on every page.

Separate the response categories from the question itself. Leave enough room in each category to comfortably respond. Cramped response categories encourage non-response. Make response categories for multiple choice questions vertical whenever possible. Horizontal groupings of response categories are more difficult to read.

When the same response categories (e.g., "yes" and "no", or “strongly disagree/strongly agree) are used for a group of questions placed together in the questionnaire, arrange the questions and the response "boxes" to be checked (or numbers to be circled) so that the response categories appear as column headings over the responses boxes or numbers.

For example:

Indicate whether you own each of the following kitchen appliances.

Yes No

dishwasher ____ ____

trash compactor ____ ____ electric knife ____ ____ Pretesting the questionnaire Most questionnaire problems can be caught in a properly conducted pretest. Unfortunately, some questionnaires are never pretested, and even when pretests are conducted, the procedures used are often woefully inadequate. Too often, those designing questionnaires feel that because they have no trouble understanding the questions, their intended respondents will have no difficulty comprehending these same questions. However, the researchers knew what the question was intended to them. An ideal pretest of a questionnaire would involve the following steps:

Match each question with the segment of the statement of information needs to which it relates. If this is difficult to do, the statement of information needs is inadequate and should be revised before proceeding.

14

Evaluate whether each individual question accurately reflects the specific information required. If this cannot be easily determined, the statement of information needs is probably insufficiently specific and should be revised before proceeding.

Those who constructed the questionnaire should try to complete the questionnaire as if they

were a respondent and someone else had prepared the questionnaire. In this regard, it is important that they read each question as if they did not already know what it was asking. They should ask themselves whether any of the questions or response categories could possibly be interpreted in more than one way. If more than one reasonable interpretation is possible, the item should be revised. Next, they should try to provide an accurate response to each question and assess whether they have any undue difficulty in doing so. In some cases the researchers will not be members of the population to be sampled. When this is the situation, they must try to identify with some member of the population of interest and try to formulate answers using the same thought patterns that they would expect their "referent" to employ. If the researchers have difficulty doing this, the researchers either have not done a sufficient amount of exploratory research or there are problems with the question and it should be revised. It is possible that the respondents will not be able to provide accurate answers to the exact questions of interest to the researchers. When this is the case, the researchers must revise their statement of the information to be collected along with the question and accept something less than they would ideally like to have.

Critical feedback from an experienced survey researcher who did not participate in designing the in questionnaire should be sought at this stage. A person of this type can often catch many of the most serious remaining problems and identify other potential problem areas for investigation in field pretests of the questionnaire. The initial field pretest of the questionnaire should involve personal interviews regardless of whether the survey is to be conducted by mail, telephone, Internet, or through personal interviews. A small convenience sample from the population of interest would be acceptable at this stage. It is extremely important that the researchers assure the subjects participating in this phase that the purpose of the interview is to detect difficulties with the questionnaire so that these problems can be corrected before the major survey is conducted. Basically, the researcher is asking these might react to the questionnaire. Depersonalizing the task in this manner makes it easier for the respondent to be critical. They don't have to be concerned about insulting the researcher or admitting that there is something in the questionnaire that they don't personally understand (i.e., I understand this question, but I think there are other consumers who wouldn't). These subjects should also be informed of the types of problems that might exist, including: 1) there might be more than one way to interpret some of the questions and/or response categories and it is possible that some subjects may have little, if any, idea what other questions mean, 2) some of the words used in the questionnaire may not be familiar to everyone in the population of interest, 3) the response categories for some questions may be inadequate (i.e., not mutually exclusive and collectively exhaustive), 4) some subjects might consider some of the questions to be insulting or otherwise irritating, 5) some of the questions may not be relevant for everyone and more skip patterns are needed, 6) some of the questions may ask for information that is very difficult, if not impossible for the subjects to provide, 7) some of the questions might ask for information that is overly personal or otherwise sensitive, 8) perhaps the questionnaire is uninteresting, and 9) the questionnaire could be too long.

Unfortunately, many researchers conduct a pretest by merely handing out questionnaires and seeing whether people will mark a response category for each question. Sometimes they will

15

tell the subjects that they are conducting a pretest and they want to find out if anything is wrong with the questionnaire. However, this rarely results in any useful information about what could be wrong with the questionnaire. People will mark response categories to the most insane questions. Furthermore, they are unlikely to mention that anything is wrong unless they are convinced that the researcher is very interested in getting critical feedback. Subjects don't like to tell researchers that there are mistakes in their questionnaires for fear of insulting them, and subjects hate to admit to researchers that there is something in the questionnaire that they don't understand because it might make them look stupid. After revising the questionnaire to eliminate the problems discovered in the initial field pretest, a second field pretest should be conducted using a quota sample representing the full range of the population. For example, if the general population is to be surveyed, this pretest should included subjects from the lower, middle and upper social classes. The same interviewing procedures would be used as those employed in the initial field pretest. When this pretest necessitates substantial revisions in the questionnaire, it should be repeated. When economically feasible, it is desirable to conduct a final pretest under the same field conditions to be used in the major survey (i.e., same method of communication; incentives, if any; etc.). No probing of the subjects regarding problems with the questionnaire would be involved. The purpose of this final pretest would be to estimate such things as the response rate, the percent of incomplete questionnaires, and the cost per interview. This step is only reasonable for very large surveys.

It is not infrequently the case that constructing the questionnaire of interest involved making only minor modifications to a former questionnaire that had been successfully used to survey a comparable sample of respondents. When this is the case, only the changes need to be thoroughly tested.