WEEK 4 PPOL 505 Exercise 3
Mz Genuine
CHAPTER6and7OSullivan.docxSelecting and Contacting Subjects
As you think about surveying clients, donors, or the general public you may start by figuring out whom to contact. Contacting everyone is seldom practical or necessary. You may use too many resources and take up too much time. Instead you will want to contact a sample of clients, donors, or residents. This chapter covers the principles guiding the selection of a sample. Knowing how to design a sample is only the beginning. You need to identify individual sample members, contact them, and encourage them to respond. Technology is rapidly changing strategies for contacting sample members and encouraging them to respond. Survey organizations are studying the impact of changing lifestyles and new communications technologies. In this chapter we focus on the basic sampling and data collection strategies. At the end of the chapter we refer you to how-to books and other sources to fill in the gaps. Our approach keeps us from overwhelming you with information that may have a short shelf life.
THE PRINCIPLES OF SAMPLE DESIGN
Sampling is an economical and effective way to learn about individuals and organizations. This is true if you seek information from individuals, case records, agency representatives, or computerized datasets. Even with a relatively small group, sampling enables you to gather information relatively quickly. Depending on how you select the sample you may be able to use the findings to generalize beyond the sampled individuals, organizations, or other units. Because sampling is a jargon-rich topic, we begin by defining some key terms.
First, you want to recognize the difference between a population and a sample. The population is the total set of units that you are interested in. A population is often composed of individuals, but it may consist of other units such as organizations, households, records, or computers. A sample is a subset of the population. Closely connected to the concepts of population and sample are the terms parameter and statistics. A parameter is a characteristic of the entire population. A statistic is a characteristic of a sample. The percentage of citizens in a city favoring a fund to finance affordable housing is a parameter. The percentage of sampled residents who favor the fund is a statistic. Depending on how you select the sample you may be able to use the statistic and the sampling error to estimate the parameter. The sampling error, which is discussed in more detail in Chapter 9 , is the difference between the parameter and the statistic and is used to estimate the parameter.
One of your first steps is to precisely define the population represented by the sample, such as “all adults over the age of 21 living in Clark County on July 1.” After you define your population your next step is to find a sampling frame. A sampling frame is a list of names of individuals or organizations in your population. A sampling frame should contain all members of your population. Complete lists seldom exist. So, in reality the sampling frame is the list of potential sample members. Sampling frames are not perfect. They may aggregate units, list units more than once, omit units, or include units that are not part of the population. A list of building addresses, for example, may include apartment buildings, but not individual units. If you plan to sample individual residences, this sampling frame would be inadequate.
The unit of analysis indicates what the data represent. You may think of the unit of analysis as what constitutes a case when you enter data on a spreadsheet. If you collect data from housing agencies, the sampling unit is housing agency. If you report the number of employees and the size of the budget the unit of analysis is housing agency. If you collect data on individual clients from the sampled agencies, then “client” would be the unit of analysis. Figure 6.1 illustrates the components of a sample.
|
|
FIGURE 6.1 From Population to Sampled Unit |
A sample design describes the strategy for selecting the sample members from the population. Designs are classified as probability and nonprobability. With a probability sample, each unit of the population has a known, nonzero chance of being in the sample. Probability samples use randomization to avoid selection biases and use statistical theory to estimate parameters. You cannot estimate parameters with nonprobability samples. Both types of designs have value but for different purposes and in different situations.
AN APPLICATION OF SAMPLING TERMINOLOGY
• Population . All participants who have completed the Clark County Job Training Program within the past 2 years
• Parameter. Average starting salaries of all graduates of the Clark County Job Training Program within the past 2 years
• Sampling frame. All (8) graduation lists from the past 2 years
• Sampling design. Probability sample
• Sample. 120 graduates randomly selected from sampling frame
• Unit of analysis. Individual graduates
• Statistic. The average starting salary of the 120 graduates was $25,000 ■
With a probability sample, you can calculate the chance that a member of the population has of being sampled. We discuss four common probability sampling designs: simple random sampling, systematic sampling, stratified sampling, and cluster sampling. These designs are often used together. Stratified and cluster samples also use simple random sampling or systematic sampling to select subjects. Multistage sampling combines cluster sampling with other designs.
Simple random sampling requires that each unit in the population has an equal probability of being in the sample. Drawing names from a hat is the prototypical example of a simple random sample. An analogous strategy is used to draw lottery numbers. You do not need to put names in a hat or marked balls in a tumbler. Rather you can use technology to help you draw a random sample. If the cases are contained in an electronic database you may use a computer program to select your sample. These programs use an algorithm to create a random sample. If the cases are not stored in an electronic database you can number the cases. In this method, using your calculator or a list of random numbers, you select a set of random numbers. Finally, you match each selected random number to the case with the same number.
If the cases are not stored electronically systematic sampling is an acceptable alternative. It is easier and normally produces results comparable to those of a simple random sample. 1 To construct a systematic sample, you divide the number of units in the sampling frame (N) by the number desired for the sample (n). The resulting number is called the skip interval (k). If a sampling frame consists of 50,000 units and a sample of 1,000 is desired, the skip interval equals 50 (50,000 divided by 1,000). You select a random number, go to the sampling frame, and use the random number to select the first case. You then pick every kth unit for the sample. In our example every 50th case would be chosen. If the random number 45 was selected the cases 45, 95, 145, and so on would be in the sample. With systematic sampling, treat the list as circular, so the last listed unit is followed by the first. You should go through the entire sampling frame at least once.
Systematic sampling has one potentially serious problem, that of periodicity. If the items are listed in a pattern and the skip interval coincides with the pattern, the sample will be biased. Consider what could happen in sampling the daily activity logs of a 911 call center. If the skip interval was 7 the activity logs in the sample would all be for the same day, that is, all Mondays or all Tuesdays, and so forth. A skip interval of any multiple of 7 would have the same result. Periodicity is relatively rare. If you notice something strange about your sample, for example, that it is all female, includes only top administrators, or has only corner houses, check for periodicity. An easy fix that often works is to double the size of your skip interval (k × 2). Go through the sampling frame once to get half your sample, then choose another random starting point and go through it a second time.
Systematic sampling may be the only feasible way to get a probability sample of a population of unknown size, such as people attending a community festival. You estimate the population size, that is, the number of attendees, determine a skip interval, and pick a random beginning point. If you plan to sample every 20th person and start with the 6th person, the 6th person to arrive (or depart) would be selected, as would the 26th person, the 46th person, and so on, until the end of the sampling period.
Stratified sampling ensures that a sample adequately represents selected groups in the population. You should consider using stratified sampling if you plan to compare groups, if you need to focus on a group that is a small proportion of your population, or if your sampling frame is already divided by groups. First, you divide or classify the population into strata, or groups, on some characteristic such as gender, age, or institutional affiliation. Every member of the population should be in one and only one stratum. Use either simple random sampling or systematic sampling to draw samples from each stratum.
Stratified samples may be proportional or disproportional. In proportional stratified samples the same sampling fraction is applied to each stratum. The sample size for each stratum is proportional to its size in the population. If, for example, you wanted to compare three groups of employees—professional staff, technical staff, and clerical staff—you would designate each group as a stratum. You would select your samples by taking the same percentage of members from each stratum, say 10 percent of the professional staff, 10 percent of the technical staff, and 10 percent of the clerical staff. The resulting sample would consist of three strata, each equal in size to the stratum’s proportion of the total population. Example 6.1 illustrates a proportional stratified sample.
The percentage of members selected for the sample need not be the same for each stratum. In disproportional stratified sampling, the same sampling fraction is not applied to all strata. Disproportional sampling is useful when a characteristic of interest occurs infrequently in the population, making it unlikely that a simple random or a proportional sample will contain enough members with the characteristic to allow full analysis. For example, a mentoring program may want to compare recruitment and retention of Spanish-speaking volunteers with that of non–Spanish speakers. You will want a sample large enough so that you can analyze and compare the two groups. If 5 percent of its volunteers are Spanish speakers, a random sample of 200 volunteers should have about 10 Spanish speakers—too few cases for analysis. Therefore, you might survey a larger percentage of Spanish-speaking volunteers and a smaller percentage of other volunteers; you could decide that half the sample should be Spanish speakers and an equal number of non-Spanish speakers, thus over-representing Spanish speakers. A higher percentage of Spanish speakers will have been sampled and non-Spanish speakers under-represented.
EXAMPLE 6.1 AN APPLICATION OF PROPORTIONAL STRATIFIED SAMPLING
• Problem. The director of volunteer recruitment for the state office of a large nonprofit organization wanted information about volunteer applicants to the organization over the previous 4 years.
• Population. All applications to the volunteer service of the nonprofit organization for each of the past selects 4 years.
• Sampling frame. Agency’s electronic file of applications(organized by application date).
• Sampling design. Proportional stratified sample with year of application as strata. A computer program select a random sample of applications for each year, using the same sampling fraction for each group ( Table 6.1 ). ■
TABLE 6.1
Proportional Stratified Sampling
|
Strata |
Number of Applications |
Sampling Fraction (%) |
Number in Sample |
|
Year 1 |
350 |
15 |
53 |
|
Year 2 |
275 |
15 |
41 |
|
Year 3 |
250 |
15 |
38 |
|
Year 4 |
230 |
15 |
35 |
|
Total |
1,150 |
|
167 |
Each strata in a disproportional stratified sample constitutes a separate sample. To conduct your analysis you must keep the samples separate as you compare the Spanish-speaking volunteers with non–Spanish speakers. To determine characteristics of all sampled volunteers you must weigh the samples. First, determine each stratum’s sample size for a proportional sample. To calculate the weight for each stratum, divide the size of the proportional sample by the size of the disproportional sample.
When you combine the samples, each Spanish speaker’s responses would be multiplied or weighted by 0.1 and each response by non–Spanish speakers would be multiplied by 1.9.
Cluster and multistage sampling take advantage of the fact that members of a population can be located within groups or clusters, such as cities and counties. Cluster sampling is useful if a sampling frame does not exist or is impractical to use. For cluster and multistage samples, we randomly select clusters and then units in the selected clusters. For example, for a study of Boys and Girls Clubs we might first select a sample of counties. Either our final sample would consist of all the Boys and Girls Clubs in the sampled counties or we would select a sample of clubs from these counties. Unlike stratified samples, which sample members from each stratum, cluster samples sample only members from the selected clusters.
Multistage sampling is a variant of cluster sampling. It proceeds in stages as you sample units dispersed over a large geographic area. The following example shows how you could design a sample to survey residents in a state’s long-term care facilities:
■ Stage One . Draw a probability sample of counties, that is, you choose large clusters containing smaller clusters.
■ Stage Two . From your sample of counties choose a probability sample of incorporated areas.
■ Stage Three . For your sampled incorporated areas obtain lists of long-term care facilities located in them.
■ Stage Four . Select a sampling strategy
■ Alternative 1 . Select all residents in the facilities identified at Stage 3 or
■ Alternative 2 . Select a sample of long-term facilities and then select all residents in the selected facilities or
■ Alternative 3 . Select a sample of long-term facilities and select a sample of residents in the selected facilities.
The units selected at each stage are called sampling units. The sampling unit may not be the same as the unit of analysis. Different sampling units are selected at each stage. In this example, the unit of analysis is residents in long-term care facility. However, different sampling units were selected at each stage of the process. Note that you can incorporate different sampling strategies into this design. For example, you may use stratified sampling at Stage One to make sure that you have counties for each of the state’s major regions, such as upstate or downstate, or the mountains and the beaches. You may use simple random sampling to choose specific residents for your sample.
Cluster sampling is recommended if your population is distributed over a large geographic area. Without the ability to limit the sample to discrete areas the costs and logistics would make probability sampling difficult, if not impossible. If site visits are needed to collect data, visiting a sample spread thinly over a wide area will be extremely expensive. Cluster sampling can help reduce this cost. Cluster sampling also helps compensate for the lack of a sampling frame. Combining cluster and multistage sampling requires an investigator to develop a sampling frame for just the last stages of the process.
Although cluster and multistage sampling methods reduce travel time and costs, they require a larger sample than other methods for the same level of accuracy. In the multistage process, probability samples are selected at each stage. Each time a sample is selected there is some sampling error; thus, the overall sampling error is likely to be larger than if a random sample of the same size were drawn.
Nonprobability samples cannot produce estimates with mathematical precision, because the sample members do not have a known chance of being selected. Nonprobability sampling is useful when designing a probability sampling is not possible, studying small or hard-to-reach populations, doing an exploratory or preliminary study, or conducting in-depth interviews or focus groups. In the following paragraphs we describe four common nonprobability designs: availability sampling, quota sampling, purposive sampling, and snowball sampling.
Availability Sampling: Availability or convenience sampling is done when cases are selected because they are easily accessed. Subjects may be selected because they can be contacted with little effort. For example, if an emergency shelter wanted to interview clients about the service availability and quality, everyone at the shelter on a given day might be invited to participate.
The obvious flaw of availability sampling is that it may exclude cases that represent the target population, and the findings cannot be generalized. The findings do not describe the knowledge, attitudes, beliefs, or behaviors of others in the target population. Consider the study at the emergency shelter. You may avoid approaching certain people. For instance, individuals who are sleeping or nodding off may not be asked to participate. These individuals may be clients of a methadone treatment program (methadone causes severe drowsiness), and their observations about drug treatment services may go unheard. If you conduct the study in the summer, the population may be very different from the winter population. The day staff may have a reputation that affects who stays around the shelter during the day.
Quota Sampling: Quota sampling, often used in market research, is less common in social science studies. Quota samples attempt to overcome the major limitation of availability samples by defining the percentage of members to be sampled from specified groups. An agency considering opening a child care center in an ethnically diverse community may want to survey local families. You could design a quota sample to ensure that you get input from families of all ethnicities. If the community is 25% Asian, 60% White, 10% African American, and 5% other, a sample of 200 should have 50 Asians, 120 Whites, 20 African Americans, and 10 from other groups. If the selection of sample members is based on convenience and judgment, they may also have characteristics that make them more approachable. This adds a potential bias to the sample.
While your sample may be representative of one variable, it is not necessarily representative of other variables. For example, your sample of families based on their ethnicity probably isn’t representative of age or income. As part of designing a quota sample you need to determine the most meaningful variable and its variation in the population. It may be easy to find information on some variables (e.g., race, gender, age) but more difficult to find out about other characteristics such as incidence of substance abuse or child-rearing practices.
Purposive Sampling: Purposive sampling selects cases based on specialized knowledge, distinct experience, or unique position. You might use a purposive sample to study very successful mentoring programs, people who underwent innovative medical treatments, or women governors. You might select cases to capture maximum variation of a phenomenon. The selected cases may provide rich, detailed information about the phenomenon of interest. Some researchers have argued that we can learn more from very successful programs and limit their samples to such programs. A variation of this theme is to compare a set of very successful programs with unsuccessful ones.
If you want to conduct in-depth interviews or focus groups you may want to select respondents deliberately. You will want to carefully select whom to interview. You will want to interview individuals who can provide insight about the phenomenon being studied, are willing to talk, and have diverse perspectives. Such individuals are referred to as key informants. Good informants will have given the subject matter some thought and can express their thoughts, feelings, and opinions.
Snowball Sampling: Snowball sampling starts by finding one member of the population of interest, speaking to him or her, and asking that person to identify other members of the population. You can use it to identify organizations or individuals. This process is continued until the desired sample has been identified. The number of members in the sample “snowballs.” Snowball sampling is most often used to contact individuals or groups that are hard to reach, difficult to identify, or tightly interconnected. For example, if you wanted to interview sex workers, you might identify a sex worker who is willing to talk to you and to identify other sex workers. In addition to identifying other sex workers, an informant may also vouch for your credibility.
A concern when using snowball sampling is that the individuals who are referred have not consented to being identified. This may not be a problem for some populations, but it is for others. Consequently, for this type of sampling, perhaps more than for others, you should remind any informant of the need to protect people’s privacy and have the informant seek permission before sharing names and contact information.
When conducting qualitative research including interviews and focus groups, determining how many cases to include in the sample may be difficult. Completeness and saturation serve as a guide for knowing when to stop selecting participants. 2 Completeness suggests that the subjects have given you a clear, well-defined perspective of the theme or issue. Saturation means that you are confident that little new information will be learned from more interviews or focus groups. Once you find that your newest subjects are sharing the same ideas, themes, and perspectives as previous participants, you can stop.
The appropriate sample size for quantitative surveys is based on several factors. The first is how much sampling error you are willing to accept, since accuracy is important in determining sample size. Greater accuracy usually can be obtained by taking larger samples. The confidence you wish to have in the results and the variability within your target population also play a role in determining sample size. Both the desired degree of confidence and the population variability are related to accuracy. We discuss these terms in more detail and their relation to sample size in Chapter 9 .
People unfamiliar with sampling theory often assume that a major factor determining sample size is the size of the entire population. That is, they may believe that representing a population of 500,000 requires a correspondingly larger sample than representing a population of 20,000. Generally, larger samples will yield better estimates of population parameters than will smaller samples. Yet, increasing the size of the sample beyond a certain point results in very little improvement and also, additional units bring additional expense. Thus you must balance the need for accuracy against the need to control costs. Further, with very large samples, the quality of the data may even decrease. Note that the relationship between sample size and accuracy applies only to probability samples. Sample size for nonprobability sampling must be governed by other considerations, such as the opportunity to get an in-depth understanding of a problem and possible solutions.
As you think about sample size keep in mind the study’s purpose. Unless its purpose is well defined and the validity of your measurements has been established, conducting a preliminary study with a small sample may be more efficient than spending resources on a larger sample.
Sampling errors come about when we draw a sample rather than study the entire population. If a probability sampling method has been used, the resulting error can be estimated. This is the powerful advantage of probability sampling. But other types of errors can cause a sample statistic to be inaccurate. Nonsampling errors result from flaws in the sampling design or from faulty implementation. To reduce nonsampling errors you must attend carefully to the selection of the sampling frame, the implementation of the design, and the quality of the measures.
Nonsampling errors are serious, and their impact on the results of a study may be unknown. Taking a larger sample may not decrease nonsampling error; the errors may actually increase with a larger sample. For example, coding and transcription errors may increase if staff rushes to complete a large number of forms. Similarly, a large sample may result in fewer attempts to reach respondents who are not at home, thus increasing the nonresponse rate.
If members of the sample who respond are consistently different from nonrespondents, the sample will be biased. Other nonsampling errors include unreliable or invalid measurements, mistakes in recording and transcribing data, and failure to follow up on nonrespondents. If a sampling frame excludes certain members of a subgroup, such as low- or high-income individuals, substantial bias may be built into the sample.
CONTACTING PEOPLE TO GET INFORMATION
You can collect data using mail surveys, telephone interviews, e-mail surveys, Web-based surveys, in-person interviews, or a combination of these. To decide which to use you may weigh time, cost, and your need for a probability sample. To get a probability sample you need to consider if you can find an appropriate sampling frame, contact sample members, and encourage their responses. Your well-designed probability sample may become a nonprobability sample if you have a poor sampling frame, are unable to contact members of the sample, or have a low response rate.
One of the first things to ask yourself is why will a subject answer your questions. You may find that clients, donors, or staff will respond if they see the value of the study and your request isn’t burdensome. The general public may not be so inclined. Survey response rates have declined substantially in recent years, which is attributed both to less success in contacting respondents and to more people refusing to participate. Low response rates, the percentage of the sample that responds, increase costs and call into question accuracy of results. Without careful planning, you risk a low response rate and wasting your time.
The next question you may ask yourself about contacting your subjects is “what will I say?” While the idea of contacting strangers to ask survey or interview questions may seem daunting, it’s actually pretty easy once you have a planned study. Much of what you will say when contacting subjects can be taken from your informed consent form discussed in Chapter 3 . When you contact subjects, the most important thing to do is to tell them who you are, what you want from them, how long it will take, and what they will get out of it. Most people are very busy but many will take the time to help you if the time seems reasonable and the study worthwhile.
In this chapter we focus on traditional data collection strategies and what you will want to consider in choosing a strategy. We do not cover the specific details of how to conduct a survey: there are excellent “how to” guides that you should consult if you plan a specific study. We spend little time on the opportunities for and challenges to data collection, such as identifying and contacting potential respondents, offered by new communications technologies. Research on new technologies is in its infancy, too soon for us to know how they can affect your work. 3 For instance, when considering a telephone survey you may want to investigate including cell phones in your sample. In addition, with the availability of inexpensive online programs to design and administer online surveys and the diffusion of Internet access, you may consider conducting an online survey, which opens up opportunities to include graphics and even video clips. As the costs and logistics of in-person interviewing climb, you may explore opportunities to conduct interviews and focus groups via video conferencing. To take advantage of new technologies and their impact may require you to keep abreast of research being conducted by survey organizations, such as the Institute for Social Research at the University of Michigan.
Mail surveys come to mind when we think about self-administered surveys. As is true of all surveying techniques, mail surveys have their unique advantages and disadvantages. All surveys require time, as you write questions, design a questionnaire, and find a sampling frame with contact information. Mail surveys consume additional time for mailing, delivering, and returning surveys. Of course, things may not go as anticipated. The survey may remain in a mailbox unattended or sit on someone’s desk. You may not know why a survey was not returned. Has the subject moved? Was he out of town for an extended period? Did she feel that the questions did not apply to her?
On the other hand, mail surveys may be less expensive than telephone and in-person surveys. You do not have to worry about finding respondents at home or in their office or contacting them at an inconvenient time. Respondents can answer the questionnaire when they want and locate any needed information. Mail surveys may work especially well if you are asking highly motivated people to answer questions about a subject that they care deeply about. An executive director may be happy to answer questions about partnerships with government—partnerships that may provide income and community status, but at the same time increase scrutiny of the agency. Employees may answer questions about benefits either to vent their feelings or in the hope that their answers will influence future management decisions.
Not everyone is willing to spend time answering a survey. Mail surveys normally have low response rates. No one is there to urge the respondent to complete it or to explain an unclear item. A respondent may put the questionnaire aside and never get around to answering it. Easily answered mail surveys should get a higher response rate. Consequently, to make mail surveys easier to complete, consider relying on closed-ended questions and ask the respondent to check a response from a list of options.
The following factors, which have been shown to affect the response rate of mail surveys, can be applied to other types of surveys.
A. Sampling Frame
1. Accuracy . An out-of-date sampling frame will reduce responses.
2. Relevance . Recipients may not respond if they believe they do not belong to the target population.
B. Questionnaire Design.
1. Length . Shorter surveys usually have higher response rates.
2. Layout and Format . The questionnaire should be easy to read and easy to fill out, with items sequenced logically. It should look attractive and of high quality.
C. Delivery and Return
1. Prenotice . Sending a message in advance to respondents telling them that they will receive a questionnaire and something about it usually improves response rate.
2. Cover Letter . Including a cover letter with the questionnaire to explain the reasons for the study, its importance, the importance of the respondent’s participation, and an offer to share results typically increases response rate. Response rates are improved if the letter includes an endorsement from someone known to and respected by the respondent.
3. Return Envelope . Including a stamped and addressed envelope markedly improves response rates.
4. Follow-up . Following up with a reminder two or more times improves response rates. Including another copy of the questionnaire with the follow-up improves rates even more. (Note: too many follow-ups may compromise voluntary participation.)
5. Incentives . Including incentives such as discount coupons, donations to a charity, participation in a lottery, or a token such as a pen can improve response rates.
A pilot test (or dress rehearsal) of the survey and its administration should identify how well the above advice works for you.
Internet surveys may be easily and inexpensively implemented. Similar to mail surveys Internet surveys should be easy to answer and return. You can use software such as SurveyMonkey, Survey-Gizmo, or Zoomerang to format a questionnaire, post it on a Web site, gather responses, and report the data on a spreadsheet. Although the software makes your work easier, the quality of the survey depends on the quality of the questions and your sample. The surveys are typically posted on the Web or sent as part of an e-mail message, so you don’t have to budget for postage or printing. You may have the data go directly into a database for later analysis. This allows you to avoid entering data from paper forms, so you don’t have to budget for data entry or verifying the accuracy of the entered data.
An e-mail message may be sent to sample members. The e-mail message should include the same information included in a mail survey cover letter and invite the recipients to complete the survey. They may be given the Web address of the online survey. Alternatively, the message may include a direct link to the survey. To control access, which is recommended to prevent multiple responses, a password or access code may be provided.
Less commonly, a survey is sent as an e-mail attachment. The respondent opens the survey, completes it, and returns it either as an attachment or through regular mail. Such e-mail surveys, however, may raise greater concerns about confidentiality, security of responses, and viruses than online Web-based surveys.
Sampling problems are particularly challenging for Internet surveys. Not everyone has equal access to the Internet and we cannot assume that a survey using the Internet will reach a representative sample. The first challenge is to find a sampling frame. Directories with e-mail addresses are available for some populations, such as members of professional associations and organization staff. This is less true of clients, customers, and the general public. If a sampling frame with e-mail addresses is available, you can choose a sample and e-mail the sample members. E-mail addresses can go quickly out of date. Sometimes a misdirected message will be returned; other times it may get lost in cyberspace. Again, respondents may delete messages from unknown sources or direct them to a spam file.
If a list of e-mail addresses for respondents does not exist, accessing the target population becomes more difficult. You may publicize the survey through List-servs, newsletters, newsgroups, and social networks. However, it is difficult to demonstrate that a true probability sample was select if a survey is disseminated through many channels. Further, you may be unable to identify the represented population. Since you do not know how many people other than those in the target population replied, you cannot calculate the response rate. On the other hand this strategy may approximate a snowball sample and help recruit difficult-to-identify populations.
If you have a sampling frame with telephone numbers you may prefer a telephone survey. It has a faster turnaround time than a mail survey and you can track the reasons for nonresponses, such as inability to contact, not part of the sample population, or a refusal. Telephone interviewing has largely replaced in-person, structured interviews as the latter has become less feasible. Interview time and travel is expensive. Both residents and interviewers may have safety concerns. Diverse lifestyles mean that people are more difficult to contact. Telephone interviewing has the added benefit of providing a rapid response from conceiving an idea through to the reporting of results. Telephone surveys allow interviewers to cover a wide geographic area, and several interviewers can work from the same location.
You can conduct the survey yourself, or you may hire a vendor. When conducting the survey yourself, remember to pretest it and call potential respondents to make sure they understand the questions and stay engaged throughout the interview. Well-trained interviewers may be effective in connecting with potential respondents, getting them to participate, and moving through the survey efficiently. They may use interviewing software with the interview questions and responses appearing on a computer screen. The interviewer reads the questions and enters the responses as the respondent answers the questions. The answers go directly into a database for analysis.
Telephone surveys have been especially valuable in collecting data from the general public. Random digit dialing (RDD) has been the preferred method for collecting data from homes with landlines. A sample is created by linking a telephone exchange with a random number; for example, a local 313 exchange could be completed by adding a random number between 0000 and 9999. RRD does not require a sampling frame and it can reach homes with unlisted numbers. However, other problems remain. Households with caller-id may not even answer. People who live in households with more than one number, which are typically more affluent than the general population, have a higher probability of being selected.
As cell phone–only households become more common, survey organizations are investigating using RDD to survey cell phone samples. One study found that cell phone interviews cost over twice as much as landline-based interviews, have higher rates of refusals, and many more ineligible contacts. 4 Interestingly enough, researchers are starting to explore address-based sampling (ABS) as a complement or as an alternative to RDD. An investigator draws a sample from a list of residential housing unit addresses and the sample receives the survey in the mail. Preliminary research has found that RDD cost 12 percent more than ABS, which included a postcard reminder and resending the questionnaire. Although ABS and RDD cannot substitute directly for one another (e.g., ABS questionnaires may have to be shorter), both approaches may be combined to collect more representative data. 5
In-person, face-to-face interviews enable researchers to obtain detailed information and ask complicated, in-depth, and sensitive questions. A few major national surveys, most notably the Current Population Studies conducted by the U.S. Census Bureau, conduct in-person surveys with a probability sample of American adults. In-person interviewing is a basic tool of qualitative researchers, who may conduct semi-structured interviews and focus groups. To arrange interviews with specific individuals a variety of strategies can be used: asking an intermediary to introduce you to potential respondents or introducing yourself through a letter, e-mail message, or telephone call. To ensure a high response rate and valid data, potential respondents are typically contacted ahead of time to schedule a time and place for the interview.
Unlike telephone interviewing, where interviewers are closely supervised and monitored, in-person interviewers are on their own. They must be well trained. They must be able to explain the purpose of the study and the meaning of the questions. They have to be able to follow the interview protocol, including knowing how to handle sensitive or confidential information. Some in-person interviews involve electronic equipment, such as video and audio recorders and computers, in which case interviewers must be able to operate the equipment and know what to do in the case of breakdowns.
The topics of sample design and data collection techniques are much more interrelated than you may have initially realized. Theoretically, designing a probability sample seems straightforward. The first challenge is identifying a sampling frame. Oftentimes sampling frames do not exist or cannot be used for the proposed survey technique. While random digit dialing seemed to resolve the problem of nonexistent sampling frames, the spread of cell phones and consumer resistance to unsolicited calls have made telephone surveying less attractive. Furthermore, telephones may not be the best medium for long surveys or ones with many open-ended questions.
A closely related problem is the response rate. A poor response rate undermines the strongest sample design. Mail surveys have generally had the lowest response rate. In-person surveys should have a high response rate, but their cost and other logistic concerns limit their use. Internet surveys should potentially get a good response rate if the survey is engaging and can be answered easily, but they are limited in their ability to reach the general population.
A third consideration is the type of questions. Mail surveys normally work best with easily answered, closed-ended questions. Although there are challenges, telephone surveys can work reasonably well with a variety of questions as long as their meaning can be easily understood. Internet surveys seem most suited for closed-ended questions, although they can also handle open-ended questions. In-person surveys are best for complicated questions, ones that require detailed or lengthy information, or ones that ask a respondent to view a video or listen to a recording.
Research is currently being conducted on how to take advantage of new technologies and databases. Some of the research, such as using ABS, may overcome the weaknesses of existing methods. Consider the possibilities of including graphics and videos on Internet surveys or to conduct focus groups using video conferencing. You could explore how residents respond to a new marketing campaign or get diverse viewpoints on how to deliver services more effectively.
Perhaps the most important lesson is to recognize that a well-designed probability sample if poorly implemented can become a nonprobability sample. Instead of a high level of accuracy, the data may have serious biases, eliminating your ability to generalize to the target population.
Arlene Fink, How to Conduct Surveys: A Step by Step Guide, Fourth Edition (Thousand Oaks, CA: Sage, 2009).
Fred Fowler, Fred, Survey Research. Methods, Sage Applied Social Science Research Methods Series, Fourth Edition (Thousand Oaks, CA: Sage, 2009).
Gary T. Henry, Practical Sampling (Newbury Park, CA: Sage Publications, 1990) is a short, readable treatment of sampling with well-developed examples from actual applied research.
Public Opinion Quarterly, the journal of the American Association for Public Opinion Research, is highly recommended to keep up to date with advances in collecting survey information.
CHAPTER 6 EXERCISES
There are four exercises for Chapter 6 . The exercises develop your competence in interpreting and applying sampling concepts and in methods to select subjects for surveys.
• Exercise 6.1 Learning from Practice asks you to find a research article and identify and assess its sampling and data collection strategies.
• Exercise 6.2 The Long Street History Museum asks you to design a sampling and data collection strategy to solicit the opinions of museum members about proposed changes as the museum deals with a loss of funds.
• Exercise 6.3 Health Care for Children and Adolescents asks you to design a representative sample from paper medical records stored in filing cabinets.
• Exercise 6.4 Own Your Own asks you to state a research question about an issue at your job or on your internship and design a sample which will help you answer the question.
EXERCISE 6.1 Learning from Practice
Locate an empirical article in a journal or on the Web. Sources for locating studies include http://www.gallup.com/poll/; http://www.ropercenter.uconn.edu; http://www.cadsr.udel.edu/default.cfm; http://www.gao.gov/).
Section A: Getting Started
1. Identify the population, sampling frame, and unit of analysis.
2. Describe the sampling design.
3. How were the data collected (telephone survey, internet survey, or some other way)? What was the response rate?
4. What are the strengths and weaknesses of the design?
Section B: Class Discussion
1. Based on your limited sample of studies (Section A) create a list “What you should say about your sample.” Include what information should be included about the
a. population
b. sampling design
c. response rate
d. sample size
EXERCISE 6.2 The Long Street History Museum
Scenario
The Long Street History Museum relies on several sources to fund its operations. These include board member contributions, donations from the public, entrance fees, membership fees, and grants from both public and nonprofit organizations. Because of a slow economy, the city will no longer provide grants unless the museum merges with another organization in the city. The museum’s executive director plans to propose two options to the board of directors: (1) merge with another organization or (2) increase fund raising efforts in order to operate without city grant money. Before doing so, however, she wants to know what the museum members think about the options. She has contacted you to help her find out what their opinions are.
Section A: Getting Started
1. The museum has over 2,000 members. You have been asked to sample 300 members to learn their opinions about the director’s proposals.
a. What questions would you want to ask the director prior to designing the sample?
b. How would you select the 300 respondents and collect information from them?
c. Justify your choice of sampling design.
2. Identify your sample’s (question 1b)
a. population
b. parameter
c. category of sample (probability or nonpropability)
d. unit of analysis
3. Identify two different methods of contacting respondents and obtaining information and discuss the advantages and disadvantages of each.
4. Develop a list of questions to ask the respondents.
5. Assume that the executive director also wants to learn what some board of directors members think about her proposals before presenting them to the full board. The board consists of 18 people who are of diverse races, ages, and professions. The board is equally split by gender. Design a sampling and data collection procedure for gathering information from board members.
6. A colleague suggests conducting interviews to explore the feasibility and impact of merging with an organization. Describe whom you would suggest interviewing. What sampling design would you use to identify and contact participants?
EXERCISE 6.3 Health Care for Children and Adolescents
Scenario
A health care organization providing specialized services to children and adolescents wants to destroy thousands of old medical records. It is required by its funding agreements with the state government to save a representative sample of medical records. The records, all in paper files, are stored in 20 metal filing cabinets. Each cabinet holds 1,000 records organized by year. The files for each year are numbered sequentially.
Small Group and Class Exercise
1. Small groups should design a probability sample using each of the following designs: simple random; systematic; stratified; multistage-cluster.
2. Each group should decide which design to recommend and justify its choice.
3. Each group should present its recommended design to the class and answer questions about its design and how the group would implement it.
4. At the end of the exercise each group should prepare a short memo evaluating the strengths and weaknesses of each design and recommend one.
1. Identify an issue related to your place of work or internship and pose one or more research questions regarding the issue.
2. Develop a sampling design for a study that would obtain information to answer the research questions. In developing your responses to the following items, use plain English to explain your methods to your supervisor and other stakeholders.
a. Identify the study population, sampling frame, sampling design, and sample members.
b. How will surveying or interviewing this sample help answer your research questions?
c. Justify your use of a nonprobability or probability sample.
d. How will the availability of resources affect your choice of design?
NOTES
1 Systematic sampling does not result in a truly random sample. Although each unit in the population has the same chance of being selected, adjacent units within the skip interval will not be selected. Systematic sampling is used widely, however, and works well.
2 H. Rubin and I. Rubin, Qualitative Interviewing: The Art of Hearing Data (Thousand Oaks: Sage, 1995).
3 A place to start exploring new technologies is Envisioning the Survey Interview of the Future, F. G. Conrad and Michael F. Schober, eds. (Hoboken: Wiley-Interscience, 2008).
4 The Pew Research Center for People and the Press, “News Release: ‘National polls not undermined by Growing Cell-Only Populations’” posted at http://people-press.org/reports/pdf/276.pdf (2006). Accessed January 17, 2010.
5 M. W. Link et al., A comparison of address-based sampling (ABS) versus Random Digit Dialing (RDD) for General Population Surveys, Public Opinion Quarterly (2008), 72(1): 6–27.
ontent of the question. For example, labeling a proposed policy as “liberal,” “permissive,” or “bureaucratic” will tend to measure respondents’ reactions to these words and not their general opinions about the subject at hand.
Rating scales can also be a source of bias. Recall from our discussion of reliability in Chapter 2 that excluding a relevant response from a list may create a bias. A rating scale with a disproportionate number of positive (or negative) ratings is also biased. For example, a list that asks a respondent to rate a service as “Excellent,” “Good,” “Satisfactory,” or “Poor” has a positive bias. Three of the four responses are positive.
Other Considerations: The best questions ask about the here and now, rather than about past behaviors or future intentions. People may not remember past behaviors; recall our comments about misreporting voting behavior. Similarly, people may not be aware of factors that will affect their future decisions. Residents who say that they would take public transportation may decide not to once it is available if it is too expensive or inconvenient, or a requested training course may prove to be less attractive once potential trainees read the description. Questions that ask respondents how much they would be willing to pay for something are particularly sensitive to wording. If the respondent feels that her answer will affect service cost, she may cite an amount lower than what she would pay. Consider a question that asks people whether they would pay $15 for a tennis license to play on the city’s public courts. A tennis player, who may be quite willing to pay $15, might try to influence a policy decision by answering “no.”
Occasionally a response list may be entirely wrong for a question. This is apt to happen if you assemble a questionnaire by cutting and pasting response lists You may inadvertently insert the wrong list. For example, you may paste the response list Strongly agree, Agree, Neither agree nor disagree, Disagree, or Strongly disagree to the question “How knowledgeable is program staff?” None of the choices, of course, answers the question.
Open-ended questions require the respondent to answer in her own words. Open-ended questions are included on surveys for five reasons. (1) They help you identify the range of possible responses. (2) They avoid biases that a list of responses can introduce. (3) They yield rich, detailed comments. (4) They give respondents a chance to elaborate on an answer to a closed-ended question. (5) They may be easier to answer than scanning down a long list of possible responses. For example, the question, “In what state do you live?” is easier to answer by writing a state name than by looking for the state on a list.
Open-ended questions are particularly helpful as a survey is being designed. The answers suggest appropriate response categories for the survey. In political polling the practice of progressing from open-ended to closed-ended questions is well established. Long before American political parties and voters express any formal preference for a presidential candidate, pollsters ask respondents to name whom they prefer as the next president. Pollsters can identify candidates with early support while avoiding the possibility that respondents simply choose a familiar name, in which case the poll would be measuring name recognition rather than support. Long before Election Day open-ended questions disappear and are replaced by closed-ended questions that ask about actual, active candidates. A similar pattern occurs if you ask stakeholders about suggested policy and program alternatives.
Answers to open-ended questions provide the rich detail that puts a mass of collected data into context. If a policy is being planned or a program evaluated, a respondent’s comment may improve an investigator’s understanding about how the public views the policy or how consumers experience a program. You may incorporate the comments into a report, which adds to its readability and keeps the interest of less quantitative readers.
Open-ended questions have three major drawbacks. First, open-ended questions require motivated respondents. Second, they complicate data compilation. Third, they increase costs, specifically the cost of training staff to collect and compile the information and the cost of verifying the reliability of the summarized information.
Respondents with little time, minimal interest in the topic, or limited communication skills may ignore open-ended questions. We assume that investigators usually have more at stake in a survey and its findings than the individual respondents. Thus, investigators overestimate respondents’ willingness to provide detailed answers. We assume that, generally respondents avoid questions that require more than just a few words to answer. This assumption may not apply if the respondent knows that the survey findings will affect a decision important to him.
Categorizing and counting the answers to open-ended questions is difficult and time consuming. If you have surveyed relatively few people, you may carefully read each answer and decide how to incorporate it in your analysis. If you have surveyed many people and don’t have time to review all the open-ended comments, one strategy you could use is to select a random sample of responses for detailed analysis. Whether you examine a sample or all completed surveys, you will find the task of categorizing the responses accurately and consistently to be challenging. To ensure reliable data, you may rescore a random sample of cases. If you give virtually the same values both times you may assume that the data are reliable. Similarly, if several analysts are categorizing responses, you should check for inter-rater reliability. Have each analyst categorize the same sample of cases and the compare categories the analysts assigned for comparability.
Beyond the development of questions, other decisions must be made to ensure that your survey is user friendly, encourages accurate and complete responses, and yields useful information. Describing recommended strategies in depth is beyond the scope of this text, but we have created a checklist to guide you in constructing a survey.
Self-Administered Surveys
■ The purpose is clearly stated in the introduction.
■ Directions on how to answer are clear.
■ Recipients who do not belong to the target population are identified, often with a preliminary question. If they are not part of the target population they are told what to do with the survey. For example, they may be asked to return the unanswered survey or to give it to a member of the target population.
■ The deadline date and return address appear on the survey.
Interviewer-Administered Survey
■ The purpose is clearly stated in the introduction.
■ Instructions tell the interviewer how to determine if a potential respondent belongs to the target population and if not how to look for a replacement respondent.
■ Directions indicate how to ask questions and record answers.
On all Surveys and Data-Collecting Instruments
■ Estimate of how long it will take to answer the questions.
■ Define critical terms.
■ Do not use abbreviations.
■ Do not use conjunctions, such as “and.”
■ Have adequate and appropriate response choices.
■ Request easily accessible data.
■ Group questions logically.
Questions on Opinion and Attitude
■ Use neutral question wording, such as “Do you favor or oppose …?”
■ Use balanced responses, for example, an equal number of positive and negative responses.
Introductions and First Questions
The introduction and first questions should engage a respondent and encourage his participation. The introduction should be short and to the point. In addition to stating the purpose of the study, the introduction should
■ identify the survey’s sponsor
■ indicate what will be done with the survey information
■ indicate whether the responses will be confidential or anonymous
■ describe how promises of confidentiality or anonymity will be maintained
■ indicate how long the survey should take
■ indicate the value of the potential respondent’s participation
Practices vary as to the need to specifically remind potential respondents that their participation is voluntary and that they may discontinue answering questions at any time. In the case of mail or telephone surveys, voluntary participation is often implied because subjects usually feel no compunction about ignoring the survey or hanging up the phone. If participation involves possible risks the introduction should identify them. The most likely risk—other than wasting someone’s time—is raising painful memories.
The following is an example of an introduction to a self-administered survey.
SAMPLE INTRODUCTION
The Oaks Chamber of Commerce wants to learn how you and others feel about the Town of Oaks as a place to work and live. If the data are to be useful, it is important that you answer each question frankly and honestly. There are no right or wrong answers to these questions.
Your answers to these questions are completely confidential. This survey is being conducted by the state university where they will be analyzed. No one other than the state university researchers will ever have access to your individual answers. So that we can follow up with nonrespondents, a number has been put on this questionnaire that can be matched with your name on a list at the university. It would be appreciated if you would leave this number intact.
Thank you in advance for your cooperation and assistance.
The introduction or the first question should ask if the respondent is a member of the target population. If not, recipients of mail surveys may be asked to return the uncompleted questionnaire. For a telephone survey, you may instruct interviewers to terminate the interview or to see if an eligible subject is available.
The best first questions are those that are easily answered and clearly related to the survey’s purpose. With face-to-face interviews, these questions build rapport between the interviewer and the respondent. In a self-administered questionnaire, the questions draw the respondent into making a “psychological commitment” to complete the questionnaire. The first questions keep a potential respondent from putting the survey aside and forgetting it; however, if the survey becomes unduly complex, confusing, or time consuming the benefits of good first questions may be lost. In a telephone survey, the first questions may be important in getting cooperation from a wary respondent who may suspect that the caller is really going to sell him something. Here are examples of opening questions.
Customer Satisfaction Survey of a Supported Employment Program:
Please rate your overall satisfaction with [name] program.
I feel like [name] program has helped me.
I would recommend [name] program to a friend.
Client Utilization Survey for a Childhood Immunization Clinic:
How did you first hear about this immunization clinic?
Is this the first immunization clinic you have attended?
How did you get to this clinic today?
Among the questions may be filter questions, which separate respondents and direct them to the appropriate parts of a questionnaire. A filter question on a housing survey might be
Do you own or rent the house in which you now live?
Own (answer question 10)
Rent (answer question 11)
Can you imagine one or more questions that apply to homeowners but not renters? For example, questions asking about mortgages, interest rates, and property taxes would apply to owners but not directly to renters.
If you plan to ask demographic questions place them at the end of the survey. By the end the respondent may be less reluctant to answer personal questions. Some investigators get into the habit of asking a standard set of demographic questions, such as questions about age, sex, race, education, and income. These questions may go unanalyzed and contribute little to the study. Surveying consumes resources, including respondent goodwill. Thus, you are well advised to stick to information appropriate to the study and to eliminate questions that have no role in the planned analysis. In addition, too many personal questions may disturb respondents who worry about the anonymity of their answers.
QUESTIONS FOR INTERVIEWS AND FOCUS GROUPS
You conduct surveys to obtain quantifiable information. You may conduct focus groups and hold interviews to get in-depth information and insights about a topic. We refer to these techniques as qualitative interviewing. Qualitative interviewing requires that you connect with your subjects, listen well, and be flexible. If you don’t have these skills you may find it more practical to conduct a survey.
Qualitative interviewing relies on the rapport and the give and take between you and your subjects. As part of building rapport and getting valuable information you must listen carefully to each answer. The answer to one question will suggest additional questions. Closely related is your ability to maintain flexibility. If your planned questions aren’t working you may have to switch gears. The beauty of interviewing is its fluidity and ability to respond to what a subject is saying.
You may erroneously assume that qualitative interviews are just chats and that you can start without having questions in mind. Prior to conducting qualitative interviews, you need to have a carefully designed set of questions. The questions should not be answerable with “yes” or “no” responses. For instance if you want to know about how an economic downturn has impacted the programs offered by a nonprofit organization, you would not simply ask “Has the recent economic downturn hindered your programs?” A respondent’s “yes” or “no” omits the detail that is the crux of qualitative interviewing. Consider asking “How has the recent economic downturn affected your programs?” Can you imagine why this question will yield more useful information? Also, note that we substituted “affected” for “hindered,” since the latter assumes only a negative impact and the interviewer would miss the opportunity to learn of beneficial impacts.
What follows are guidelines that you will want to keep in mind as you write questions for qualitative interviews.
Is This Question Necessary? Try to stay on task when conducting an interview. Be sure to remember your research question as you write questions and potential probes. Remember, participants have agreed to spend their valuable time with you. Asking questions for the sake of asking questions wastes their time. Therefore, ask yourself: Is this question relevant? Will it help answer the research question? Will it add to my knowledge of the issue at hand? Will it explain any theories or hypothesis I am exploring? Is it simply satisfying my curiosity? Keep in mind that each question adds time to the interview, the transcription, and the data analysis. Each task is time consuming.
Is This Question on Topic? Each question should be related to the research question. While interviewing, you may get off topic—especially if you have a good rapport with your respondent. We are all social creatures and most of us love to talk about ourselves. Therefore, remind yourself of the importance of staying on topic! Remember why you are there and what you are doing.
How Will This Question Be Heard? Remember your subjects. Individuals have their own respective lenses through which they view the world. Commonly, when people are asked why they perform a certain act, they may hear it as a challenge to their decision. A question like “Why did you stop being a vegetarian?” may be heard as an attack on the decision. Therefore, imagine how others will interpret your questions. Just as we urge you to pilot survey questions, be sure to pilot interview questions. Are they clear? Do they get the responses you seek? Does a question prompt the respondent to go into a completely different direction or become defensive? Returning to the example regarding vegetarians, when questioning former vegetarians, a less confrontational question might be, “Why did you decide to include meat in your diet?”
Is This Question Leading? Social desirability is an even greater concern with qualitative research than with quantitative research. You are sitting right there looking at the individual as she answers. That’s pressure! The more the questions and the way they are asked can be kept neutral, the better. If you imply how you would like respondents to answer, it is more than likely they will answer that way.
Other Considerations: As previously discussed, open-ended questions generate valuable information. However, even more detail may be necessary to get a clear understanding of the response, so you may want the respondent to elaborate. This is where probes are useful. Probes are questions meant to get more detailed answers. Hypotheticals, posing the ideal, and acting as the devil’s advocate are three types of probes that you may find useful.
■ Hypotheticals pose a suppositional circumstance, condition, scenario, or situation. For example, you might ask, “Suppose your organization lost funding for the credit counseling program. What strategies would your organization use to recover?” Such questions encourage the respondent to think beyond what currently is, to what could be.
■ The ideal or the magic wand asks respondents to think about the perfect scenario. For instance, the mission of many social welfare organizations is to correct a social ill. A probe could ask what is needed to accomplish the mission. You might ask a credit counseling organization, “What has to happen to make sure that all homeowners have the necessary information to avoid foreclosures?” Alternatively, you might ask the respondent what would change if he could “wave a magic wand.” You might ask, “If you could wave a magic wand to avoid foreclosures what would you change?” Both strategies provide respondents with the opportunity to think beyond their current situation and explore additional options.
■ Devil’s advocate probes put you in the position of arguing against a cause or position. You should not play the role of a committed opponent, but someone trying to understand the respondent’s cause or position. It is important to remain nonconfrontational. Typically, interviewers introduce such probes by saying, “Let me play the devil’s advocate.” Acting as the devil’s advocate allows respondents to consider their responses, justify their current position, fill in missing pieces of information, or think through other options. An example of a devil’s advocate probe is “Let me play the devil’s advocate and ask why homeowners who knowingly took on too much debt should avoid foreclosure?”
In addition to designing questions you need to think about the interview itself. Two topics worthy of consideration are how you are going to treat silences and handle vague responses.
Your parents may have told you that silence is golden. In the case of interviewing they are right. Often new interviewers are uncomfortable with the silence that follows a question. You may think the respondent does not understand the question. However, often, silence may indicate that a question is particularly poignant or that you have raised a troubling or controversial issue. Silence allows respondents to organize their thoughts and give thoughtful answers. Jumping in with a rephrased question may encourage a superficial answer or one that lacks depth the respondent was pondering. Qualitative interviewers must become comfortable with silence. Silence is your friend in the interview process.
Seeking concreteness is another important technique. Often respondents will answer in general terms. For qualitative research to be meaningful, concreteness and specificity must be achieved. A surefire sign that you are not dealing in specifics is if respondents use words such good, bad, okay, many and do not explain why something is good or what is meant by many. The terms will not provide the information needed for adequate analysis. When answers aren’t concrete follow up with questions to find out what is meant or intended. Does “good” mean beneficial, adequate, entertaining, or something else? For instance, if a housing program director suggests that the program is doing “okay,” does he mean that the program is breaking even? meeting client needs? well liked by the community? at maximum use capacity? meeting or exceeding its goals? fully staffed? What exactly is “okay” about the program? In addition, remember that one person’s “okay” is another’s “good” and another’s “less than we would like.”
PLANNING, PRETESTING, AND PILOTING SURVEYS AND INTERVIEW PROTOCOLS
Relevant questions require systematic development. The first stage is developing questions. Before the questions are drafted the critical stakeholders should identify the information they want and agree that a survey, interview, or focus group will produce the desired information. As the questions are drafted you may want to meet with these stakeholders again to confirm that you are asking the “right questions.” You should review the drafted survey and interview protocols with colleagues and ask them: Are any items ambiguous? Are the response choices appropriate? the information easily available to respondents? If their answers indicate that the questions are unambiguous, with appropriate response choices and accessible information, you have qualitative evidence that the questions are reliable. Writing questions takes time and effort; you need to have a thick skin. Good questionnaires and interview protocols may emerge only after criticism and argument. If you have already put considerable time and thought in developing the items, you may automatically disagree with colleagues’ criticisms. In our experience, reviewers often identify the same items that actual subjects find troublesome.
After the questions are written and reviewed, the next step is to pretest the draft survey by asking a few people, similar to the prospective respondents, to answer it. For example, an employee survey should be given to a few employees ranging from unskilled laborers to senior management; they may also represent various divisions. A pretest normally involves face-to-face contact with the participants. You should record how long it takes individuals to answer the survey and if it seems to hold their interest. If a survey does not engage respondents, the information may be less accurate, incomplete questionnaires may be more common, or, in the case of mailed questionnaires, the response rate may be lower as respondents discard or ignore the survey.
Qualitative interview protocols also need to be pretested. Conduct interviews with several different people. Estimate how long the interview will take if a respondent answers all the questions. Note and revise questions that get respondents off topic. Document where you may need additional probes or should allow for silences. An interview that does not generate answers to your questions will leave you with lots of information but little that is relevant to your study. You will have wasted time, money, and resources.
At the end of the pretest consider asking the respondents for their reactions to the questions and the survey or interview as a whole. Also, use your own observations to identify potential problems. Are directions clear? Is the question order logical? Pretests commonly result in the investigators changing or clarifying unclear terms, rewording or dropping ambiguous questions, changing response categories, and shortening the survey or interview.
The last step is the pilot study or dress rehearsal. The study is implemented as planned on a small sample representing the target population. The planned analysis is carried out on the returned surveys and interview transcripts. The dress rehearsal should identify problems in contacting members of the sample and in getting their cooperation. The time involved in collecting and compiling the data can be estimated. The feasibility of the planned analysis can be verified. Previously unreliable, or invalid measures may be detected. Unused measures may be dropped. The investigators will especially want to note insensitive questions or questions that yield a large number of “Don’t know” responses. Problems encountered in the pilot study should be resolved prior to implementing the survey.
In studies with a limited budget or a short timeline the pretest and the pilot study may overlap. At a minimum you should test the instrument or interview questions on potential subjects, check that items have sufficient variation, and conduct basic analysis to assess whether the survey will produce useful information.
Without conducting a pilot study and checking the proposed analysis, resources can be wasted. Consider an actual survey that resulted in 33,000 responses. The program staff spent a few frantic weeks simply compiling the responses to the 21 closed-ended and 2 open-ended questions; the total time spent just compiling the data was 550 hours. The data analysis was less costly, because the questions didn’t allow for much analysis beyond summarizing the responses to each question. (Because of ambiguous wording all responses to two questions had to be discarded.) And remember, the time and cost investment would have been much greater had the data collection involved intervews. To conduct interviews and transcribe the data only to find that most of the conversations were off topic would be a devastating blow to any researcher!
You may have begun reading this book assuming that quantitative research and qualitative research had little in common. We hope that by now you have observed that their sampling strategies, data-collection methods, and questioning techniques overlap. Both types of research require a defined research question that can be answered by the questions you ask and how you ask them. Both attempt to balance efficiency with obtaining quality information.
An oft repeated theme was the need to pretest questions and to obtain feedback from potential respondents and stakeholders. Another theme to keep in mind is making sure that questions don’t encourage biased responses, socially desirable responses, and self-serving, and defensive responses.
To keep up to date with current research on survey research topics such as question wording, questionnaire design, data collection, and data analysis, see Public Opinion Quarterly, the journal of the American Association for Public Opinion Research, published by the University of Chicago Press. Also see Recommended Resources listed at the end of Chapter 6.
Fowler, F. J., Improving Survey Questions (Thousand Oaks, CA: Sage Publications, Inc., 1995).
Fowler, F. J., Survey Research Methods, Fourth edition (Thousand Oaks, CA: Sage Publications, Inc., 2009).
Seidman, I., Interviewing As Qualitative Research: A Guide for Researchers in Education and the Social Sciences (New York: Teachers College Press, 2006).
CHAPTER 7 EXERCISES
There are three exercises for this chapter. The exercises develop your competence in writing and assessing questions and questionnaires.
• Exercise 7.1 The Long Street History Museum (Revisited) asks you to write questions for the Long Street History Museum study, to review them with a few classmates, and then identify in a class discussion qualities of good and bad questions.
• Exercise 7.2 Metro Citizen Kitchen and Food Pantry asks you to write questions and assess questions for the organization to use to routinely survey its clients.
• Exercise 7.3 On Your Own asks you to apply a checklist to a questionnaire that you may have been asked to develop as part of an internship or your job.
EXERCISE 7.1 The Long Street History Museum (Revisited)
Scenario
In exercise 6.1 the director of the Long Street History Museum proposed two options to the board of directors so that the museum could continue to receive grants. The options were to (1) merge with another organization or (2) increase fund-raising efforts in order to operate without city grant money. Before doing so, however, the director wants to know what the museum members think about the options.
Section A: Getting Started
1. Write questions to ascertain the members’ opinions about the two options.
Section B: Class and small group exercises
1. Work with one or two classmates to critique each other’s questionnaires.
2. Participate in a class discussion where you
a. identify common errors in question writing
b. develop a list of examples of “good” questions and “poor” questions
EXERCISE 7.2 Metro Citizen Kitchen and Food Pantry
Scenario
Mohan Greene is the newly appointed executive director of the Metro Citizen Kitchen and Pantry (MECKP). The organization distributes food received from food drives, restaurants, and supermarkets to seven community soup kitchens and food pantries. Last year the kitchens served 13,000 meals, and pantries supplied food bags for 480,000 meals. (A food bag contains groceries adequate for three lunches or dinners.) Mohan is considering routinely surveying soup kitchen and food pantry users to see if the program is reaching its target clientele.
Section A: Getting Started
1. Write a closed-ended question that is
a. a factual question
b. a knowledge question
c. a behavior question
d. a motivation question
e. an attitude or opinion question
2. For each question you wrote above indicate how Mohan might use the answers to improve MECKP.
3. Use a search engine to find a survey or survey questions for users of soup kitchens or food pantries. Use the information in this chapter and assess the introduction, question order, question, and response wording.
4. Use qualitative methods (see Chapter 2) and assess the reliability of the questions.
5. MECKP has a survey it plans to administer at the soup kitchens.
a. Why should it do a pilot test?
b. Write a list of tasks that need to be done as part of the pilot test.
Section B: Class simultation of drafting a questionnaire for MECKP
In groups of three to five draft a short survey and introduction for MECKP to administer to residents in its service area. It should have no more than 2 open-ended and 15 total questions. The questionnaire is to identify people’s knowledge of what constitutes a good diet. MECKP plans to use the information to guide its development of an educational campaign.
1. Outline a short presentation that you would make to MECKP identifying what it may learn from the survey questions.
2. Make your presentation to the class and briefly review your questionnaire.
3. The class will select a questionnaire, make needed edits (improve question wording, response lists, and similar changes) and prepare for pretesting.
4. Each member of the class should administer the selected questionnaire to five people. Note how long it takes them to respond and any questions that they find confusing, unclear, or otherwise flawed.
5. Bring your completed questionnaires to class.
a. Review each question and decide if it needs to be revised or deleted before including it in the final questionnaire.
b. Compute the frequency of the responses to questions that you plan to use in the final questionnaire.
c. Based on the frequencies does the class think that each question has sufficient variation to be worth keeping?
Put the completed questionnaires aside for analysis after reading Chapter 8.
If you have been tasked with developing a questionnaire, it may seem daunting. As discussed in this chapter, there are lots of considerations and choices to be made. Apply this checklist as you design your questionnaire. Once you are finished, review your questionnaire with your supervisor or other colleagues. Write a short memo assessing how well the checklist worked.
□ Determine the purpose of your questionnaire.
□ Learn what questions stakeholders want answered.
□ Determine how stakeholders plan to use the answers.
□ Determine the type of questions to ask.
□ Do I need behavioral questions?
□ Do I need factual questions?
□ Do I need knowledge questions?
□ Do I need opinion questions?
□ My questionnaire has an introduction.
□ My first questions are easily answered and clearly related to the survey’s purpose.
□ Questions are worded for optimal responses.
□ Questions are worded so that a respondent understands what they mean.
□ Questions are worded so that a respondent answers honestly.
□ I have defined any unfamiliar terms.
□ Questions specific about the time, place, or amount are included.
□ Where appropriate, I have offered Don’t know as a choice to discourage uninformed responses.
□ Some statements go in different directions.
□ Questions are stated so that they suggest to the respondent that any possible answer is acceptable
□ Questions are free of bias.
□ Loaded questions, which have words or phrases that evoke a strong positive or negative response, have been omitted.
□ If I am using open-ended questions
□ I am confident that respondents will be motivated enough to answer them.
□ I will be able to manage the additional data compilation.
□ I have the resources needed for the increased costs, specifically the cost of training staff to collect and compile the information and of verifying the reliability of summarized information.
NOTES
1These statements are from an extensive measure of what motivates food choices and can be found in A. Steptoe, T. M. Pollard, and Wardle, “Development of a Measure of the Motives Underlying the Selection of Food: The Food Choice Questionnaire,” Appetite (1995), 25:267–284.
2Most of the information on validating voting data is from S. Presser and M. Traaugott, “Correlated Response Errors,” Public Opinion Quarterly (Spring 1992), 56:77–86. For a discussion on voter records as a source of error, see P. R. Abramson and W. Claggett, “The Quality of Record Keeping and Racial Differences in Validated Turnout,” Journal of Politics (August 1992), 54:871–880.
