Research methods unit V Power point and DQ question

conducting_research.pdf

Home >Business & Finance homework help >Management homework help >Research methods unit V Power point and DQ question

Statistical Journal of the IAOS 26 (2009) 1–14 1 DOI 10.3233/SJI-2009-0692 IOS Press

Conducting research on the response process in business surveys

Mojca Bavdaž Faculty of Economics, University of Ljubljana, Kardeljeva pl. 17, SI-1000 Ljubljana, Slovenia Tel.: +386 1 5892 630; Fax: +386 1 5892 698; E-mail: [email protected]

Abstract. Measurement errors in business surveys can be a serious obstacle to achieving accuracy in economic data, yet knowledge about such errors is relatively modest. It is thus not surprising that approaches to gain this knowledge remain inconsistent and have been rarely discussed. Research into the response process has proved beneficial in revealing and evaluating the sources of measurement errors in surveys of individuals. This article proposes a blueprint for conducting research into the response process in business surveys in line with modern quality thinking. It analyzes the methodological and practical challenges of such research and illustrates them with an empirical study of the response process in a business survey conducted in cooperation with the Statistical Office of the Republic of Slovenia.

Keywords: Business survey, measurement error, qualitative research, response process

1. Introduction

Economic data have always been important for the successful performance of governments, enterprises, and other institutions, yet their relevance, accuracy, and timeliness have never been so much in the spotlight as they are in the information society. Accuracy is a re- quired characteristic of data quality, and all other quali- ty dimensions are judged against accuracy [25]. In busi- ness surveys, error reduction has traditionally focused on sampling errors, frame errors, and nonresponse, and to a lesser extent on measurement errors [47]. On the one hand, fewer measurement errors have been expect- ed in business surveys because respondents are con- sidered more competent and well equipped with busi- ness records to perform the survey task. On the other hand, the progress of sampling theory occupied most of the methodological development during the first half of the twentieth century [31]. Quality frameworks of the most prominent statistical organizations acknowl- edge the importance of measurement errors, but meth- ods for assessing measurement errors and information on measurement errors required for quality reports are still inconsistent.

Most of the research exploring the sources of mea- surement errors in business surveys has been tentative or hypothetical rather than based on actual data collec- tion, which is consistent with the relative abundance of pretesting and the scarcity of quality assessment re- search. In addition, research is often restricted to an ex- pert evaluation of the survey questionnaire [47]. Rare on-site visits are usually dedicated to the pretesting of a new or redesigned questionnaire, which may also be limited to a subset of survey questions. It may also be difficult to reliably replicate the research because all the necessary protocol details are rarely revealed in published research. As a result, knowledge of mea- surement errors in business surveys is modest despite its significance.

This article systematically analyzes the challenges of designing and implementing a qualitative study of the actual response process in a business survey that aims to reveal and evaluate the sources of measurement errors. Its focus is on developing the methodology of such a study. It discusses the key issues concerning the methods of data collection, the selection of study units, the design of protocols, the logistics of data collection, and the methods of data analysis.

2 M. Bavdaž / Conducting research on the response process in business surveys

These issues are illustrated here with an empirical study carried out in 2005 that aimed to shed light on an actual response process from start to finish in a typi- cal business survey, with emphasis on the business per- spective to identify and evaluate potential sources of measurement errors. The business survey under study was the Quarterly Survey of Trade (QST) conducted by the Statistical Office of the Republic of Slovenia (SORS). In terms of generalizability to other business surveys, this survey is an appropriate candidate for ex- amination because it was a mandatory recurring survey conducted by a governmental organization. The sur- vey instrument mainly collected quantitative business data on sales, stock (i.e., inventory), and employment to track changes in totals and structures. Data col- lection was self-administered and conducted by mail. Nevertheless, survey staff occasionally came in direct contact with businesses, such as during nonresponse follow-up or editing. The editing staff first checked the returned questionnaires manually for missing data and major inconsistencies. After a manual double entry of numerical survey data, computerized edits were carried out and some businesses were contacted to clarify ma- jor inconsistencies or to confirm unexpected respons- es. Finally, subject-area specialists analyzed the aggre- gate data and verified any remaining critical deviations that had a significant impact on the aggregate estimates before the release (for more detail, see [4]).

Design and procedures of the QST influenced the methodology of the empirical study of the QST re- sponse process. This article argues for the decisions made in conducting the research and discusses validity issues that are especially pronounced at this early stage of systematic research into the response process in busi- ness surveys. It concludes with recommendations for survey work and future research.

2. Methods of data collection

Several methods pertaining to the qualitative re- search tradition can be used to examine the response process and the potential sources of measurement er- rors in a business survey. Data can be collected by con- ducting an expert evaluation, by observing the process, or by interviewing the people involved in the process. Previous research has shown that the methods are usu- ally complementary; in spite of the overlap, one method cannot fully substitute for another because each has its own strengths and limitations [17,36]. It is therefore best to find a suitable mix of methods within the avail-

able resources and time frames. This section first pro- vides an overview of the most common methods ade- quate to the study of the response process in business surveys and then describes the empirical study on the QST response process.

2.1. Scope of expert evaluation

According to Forsyth and Lessler [15], expert eval- uation methods do not involve interaction with respon- dents and include behavior coding, expert analysis, and cognitive forms appraisal methods. Behavior coding is suitable only for interviewer-administered surveys, while an expert analysis (or expert review, desk review, or paper review) does not depend on the survey mode. It focuses on the questionnaire, indicating potential sources of measurement errors in the survey. Experts may belong to four different categories: questionnaire design experts, questionnaire administration experts, subject-matter experts, and computer-based expert sys- tems [48].

Questionnaire design experts typically focus on the wording, structure, and order of questions; the re- sponse alternatives; the navigational rules of the ques- tionnaire; and instructions given to interviewers [22]. They may be assisted by computer-based expert sys- tems [21]. However, business surveys often use la- bels instead of questions, their terminology is closely related to the contents, and the requested information is complex and technical. Therefore, it may be nec- essary to evaluate the questionnaire layout separately from its contents, which is the domain of subject-matter experts. The evaluation of the questionnaire layout can use the design principles for self-administered ques- tionnaires developed and promoted by Dillman and his colleagues [12,26,39]. As Ramirez suggests [38], when consulting subject-matter experts, we should preferably use a combination of specific questions on survey items and a well-structured schedule so as to avoid the risk of a misplaced focus. He also cautions against the lack of depth in an expert’s commentary, as some experts may confine themselves to problems.

Questionnaire administration experts typically in- clude experienced interviewers. Interviewer debrief- ing can provide useful insights into the response pro- cess. However, interviewers rarely administer busi- ness surveys. Questionnaire administration experts are thus often overlooked in business surveys, although direct contacts with business respondents occur, for instance, during respondent selection, telephone re- minders, and follow-up. In fact, Rowlands, Eldridge,

M. Bavdaž / Conducting research on the response process in business surveys 3

and Williams [40] suggest using interviews with data- editing staff as a cost-effective means of questionnaire evaluation. They hold that data editors obtain a useful insight into the business perspective on the question- naire’s completion, though their contacts are confined to somewhat problematic businesses that failed the val- idation check and they may lack an adequate under- standing of the underlying issues because of inadequate training.

Although in practice an expert analysis may be the only evaluation of a questionnaire, the procedures and outcomes of expert analysis have not been well doc- umented [47]. To standardize an expert review in es- tablishment surveys, questionnaire appraisal (or cog- nitive forms appraisal, questionnaire appraisal coding system, or expert appraisal) may be used [16]. With this type of questionnaire evaluation, codes are developed on the basis of a survey response model and assigned to question features that are likely related to measure- ment errors [29]. Standardization can be particularly helpful in comparisons, but it may reduce the openness to unexpected problems.

Evaluation of an ongoing business survey may also incorporate a systematic review of completed forms, which consists of examining errors, handwritten com- ments, crossed-out words and numbers, and so on, which provide information on the response process such as the presence of multiple respondents or mis- leading design (see, e.g. [20]). This is a relatively cheap and easy evaluation method for paper self-administered business surveys, as it can be done any time after the survey and places no burden on businesses.

2.2. Feasibility of observation

The most straightforward method for studying the response process is to observe it as it is happening. De- spite the reliability of structured observation based on a predetermined schedule, it may be more advantageous to be open to the discovery of unexpected elements [3]. Observation as a stand-alone method was not widely practiced in business surveys, because of their preva- lent self-administration mode, until the advent of Web surveys, which are exposed to usability testing. Be- cause business surveys usually involve the retrieval of data from records or from other people, conducting an observation of the response process in a laboratory set- ting is unreasonable. In such a case, the only option that will provide any insight is in situ observation of the response process [20,24]. It is technologically pos- sible to capture interactions with records or between

people in their entirety: video recording and computer data logging may produce the best type of field notes, with every detail perfectly registered. Yet management may view such documentation as an unacceptable intru- sion into daily activities or as violating confidentiality. Moreover, the additional cost of such documentation may be nonnegligible and the abundance of collected data may reduce any immediate insights. As a result, observation alone usually offers only partial insight.

Conducting observation has disadvantages with or without technological support. One of them is the usual unfamiliarity with the file storage system or software, which calls for complementary methods such as inter- viewing. Another unavoidable drawback is potential reactivity. It is important to understand how the ob- server (and the recording device) affects the situation and which factors contribute to the observation, as ob- jectivity is illusory [1]. Last but not least, observation is time consuming.

2.3. Capacity of interviewing

The purpose of interviewing is to collect informa- tion directly from the people involved in the process of responding to a survey. Debriefing respondents in a response analysis survey is inappropriate if we want to follow the exploratory path. A better option is to adopt a qualitative research interview.

Previous research on the survey response process in surveys of individuals relied heavily on cognitive in- terviewing. Although various techniques can be found under the umbrella of cognitive interviewing, they all produce verbal reports [10]. The verbal material be- yond a simple survey response is then used to eval- uate whether the questions are capturing information as intended [5]. Two techniques are most commonly associated with cognitive interviewing: thinking-aloud interviews and verbal probing.

In a thinking-aloud interview, respondents are en- couraged to verbalize their thoughts while answering the survey questions [49]. As Ericsson and Simon ar- gue, the main characteristic and strength of thinking- aloud interviews is that they minimize the elapsed time between the inception of a thought and its reporting, and so they capture thoughts in the respondents’ short-term memory that may otherwise be lost (as cited in [15]). Because the interviewer intervenes minimally, the in- formation provided may be unexpected and thus espe- cially valuable [48].

However, the task of answering questions in a recur- ring business survey does not possess those character-

4 M. Bavdaž / Conducting research on the response process in business surveys

istics that have been cited in the literature as enhanc- ing the validity and efficacy of the thinking-aloud tech- nique, namely, being verbal, novel, interesting, and en- gaging; having an emphasis on problem solving and de- cision making; and involving conscious, higher-level, long-enough verbal processes [48]. It can be assumed that the recurring character of a survey brings rou- tine and automatic unconscious processing into the re- sponse process, which means that respondents do not think a lot or enough about the task and retrieve some information directly from memory. The task is not completely verbal; it requires the processing of numer- ical, symbolic, and graphic information in addition to the reading or scanning of a considerable amount of written text as well as interacting with business records. The only promising feature of the task is that it empha- sizes problem solving and decision making. Therefore, it seems only appropriate to use such interviews with those people who are to fill out the questionnaire for the first time.

The other common variety of cognitive interviewing, verbal probing, presents itself as a suitable approach for people who are already participating in the survey. Ver- bal probing is administered after the response has been provided. Interviewers tend to use expansive probes designed to obtain additional narrative information and confirmatory probes to check their understanding of the respondent’s answers; however, traditional cogni- tive probes to induce interpretation of terms, thought processes, and answers are used modestly [5,6]. In fact, Gerber [18] proposed using in-depth interviews separately or immediately after cognitive interviewing to collect information on how respondents use concepts and terminology. This ethnographic interviewing pro- vides a means to extend our understanding of the phe- nomenon under study with the respondent providing supplemental background, cultural, and other contex- tual information.

A specific approach designed to test self-administ- ered questionnaires, which business surveys usually are, is the Three-Step Test Interview, which combines thinking-aloud and verbal probing (see [23]) to best ex- ploit the comparative benefits of both techniques. Such interviews consist of first observing the actual comple- tion of the questionnaire with the help of concurrent thinking aloud, conducting a focused interview to rem- edy any gaps from the first part, and finally carrying out a semistructured interview to elicit experiences and opinions.

Another form of interviewing is the use of vignettes, as Stettler, Willimack, and Anderson suggest [42]. Vi-

gnettes elicit a respondent’s potential response to hypo- thetical situations while also revealing the conceptual boundaries of the domain [9]. They are most practical and efficient in the case of intensive targeting of a few identified key comprehension issues [48]. Reinterview or post-enumeration studies to identify measurement errors and reconciliation studies to explain them are not a viable option in most business surveys because respondents generally keep a copy of the completed questionnaire. In business surveys where the response process includes several people, group interviews can be an efficient way to collect information from a par- ticular business (see [43]). However, the method may not be the most efficient from the business perspec- tive because of the different roles, and it may reduce participants’ openness. Focus groups have an excel- lent place in the development of multiple meanings and perspectives, and in unveiling tacit norms and norma- tive assumptions [27]. However, their application to respondents in business surveys is limited because of difficulties in setting and respecting meeting dates [41]. They also may be inappropriate to tackle response pro- cess issues because of memory loss and confidentiality issues.

2.4. Methods used in the study of the QST response process

Selection of methods took into account the relative effectiveness and efficiency of the methods as well as the available resources and time frames. I decided to use a mix of expert evaluations, observations, and interviewing methods to shed light on the QST response process from diverse perspectives. This required the development of a range of approaches.

The expert evaluations relied on the insights from representatives of all of SORS’s survey-related activi- ties (people involved in activities after fielding the ques- tionnaire and subject-area specialists) and two account- ing experts with an economics background. Data were collected through interviews, except for evaluation of the questionnaire layout, which was carried out sep- arately as a desk review. Although the questionnaire appraisal was not formally used in the QST study, its coding system was consulted before designing the in- terview protocols. The semistructured interview with the SORS representatives aimed to acquire information on the conduct of the survey, its development over time, and problems encountered in the past.

The protocol for subject-matter experts was de- veloped separately in line with recommendations by

M. Bavdaž / Conducting research on the response process in business surveys 5

Ramirez [38]. To capture the most information, the experts were asked to think aloud when commenting on the questionnaire item by item and to focus on ques- tion meaning, question clarity, and data availability in businesses. Afterward, they were asked to assess their navigation through the questionnaire. After this first part of the interview, the second part of the interview tried to clarify any unclear comments, follow up on in- teresting ideas, probe those survey questions that were already considered problematic, and related conceptual issues.

Another method used in the QST study was in situ observation. The method was selected despite that it is time consuming, because it may provide some detail about the response process and some information that other methods may not capture – because that informa- tion is, for instance, perceived as irrelevant, unimpor- tant, or self-evident. However, in situ observation was used only in a few cases with a relatively uncomplicated response process and without video recording, because of limited resources and the comparative efficiency of other methods. Its extent depended on the amount of data that respondents had retrieved before the visit.

The qualitative research interview was selected as the primary method of investigation in the business- es. The interview for people already familiar with the QST questionnaire mainly relied on retrospective prob- ing and ethnographic interviewing. The interview for people new to the QST questionnaire was based on the Three-Step Test Interview. Although vignettes were not planned because the study focused on an actual re- sponse process rather than hypothetical situations, the idea of hypothetical probing was retained and used with items considered nonapplicable to the business.

The research design could have been improved with examination of completed QST questionnaires, but the first round of manual editing made it difficult to reliably distinguish between respondents’ and editors’ input.

3. Selection of study units

3.1. Selection of businesses

The selection of businesses aimed to cover the het- erogeneity of response processes; it did not aim to con- struct a representative sample from which to generalize on a statistical basis. Business size may be defined as the single most important business characteristic that is assumed to influence or be related to the characteristics of the response process [35]. Because business size is

related to the specialization of work and the fragmen- tation of knowledge [44], it can be expected that larger businesses have more complex response processes than do smaller ones and a greater variety in the response process because of various organizational issues. It was therefore sensible to opt for businesses of different sizes and to include several larger businesses.

There is no standard procedure for determining sam- ple size in qualitative research. One approach is to con- tinue collecting data until no new information emerges and the saturation of categories is reached, which has been criticized because it is again difficult to justify [8]. In exploratory interview studies, the sample sizes tend to be small, “around 15 ± 10” [28]. The number of units in previous related studies varied considerably (see Table 1).

In the QST study, a sample of 25 businesses was selected systematically across all business sizes from the QST panel. After the sample was selected, it was verified that the sample well covered the geographical location as well as the various combinations of eco- nomic activities (trade as the only, main, or secondary activity), trade activities (retail, wholesale, and com- mission), and merchandise (e.g., food and beverages, textiles, furniture, books, motor vehicles, fuels). These variables may be related to the survey response pro- cess through different regulations, taxation, competi- tion, and other aspects of the business environment. Responding early, on time, or late with regard to the deadline may also indicate some characteristics of the survey response process, such as data availability, so the usual timing of survey response was checked in the first telephone contact with the businesses to be sure to have representatives of all businesses in the sample.

The initial sample included none of the largest play- ers in the Slovenian trade. This was the reason for the second round and the purposive selection of seven businesses that were among the largest in a particu- lar trade activity or in trade as a whole. Furthermore, a request was made to the SORS to randomly select ten businesses that had not already completed the QST questionnaire.

3.2. Selection of research subjects

Once the businesses were selected, it was necessary to determine who to include in the QST study if more than one person participated in the response process and when to use observation as a method of inquiry. The request for an on-site interview was addressed to the person with the main role in completing the QST

6 M. Bavdaž / Conducting research on the response process in business surveys

Table 1 Number of units in previous studies related to the response process in business surveys

Study reference Number of units

Babyak et al. (2000) 2 to 15 interviews, and exceptionally 50 interviews, for testing industry-specific business questionnaires Edwards and Cantor (1991) 24 semistructured in-person interviews about the record-keeping system and questionnaire completion Eldridge et al. (2000) 35 in-depth interviews for testing a business questionnaire Giesen and Hak (2005) 11 on-site visits with observation and debriefing interview, and 12 on-site retrospective focused interviews Hak et al. (2003) 5 and 24 on-site visits with a combination of observation and focused interviews about the response process

and the response burden in two surveys O’Brien (2000) 8 cognitive interviews with business respondents O’Brien et al. (2001) 5 cognitive interviews with business respondents Stettler et al. (2001) 17 and 40 cognitive interviews for testing two questionnaires Sudman et al. (2000) 30 group interviews about the response process in business surveys Willimack (2003) 31 and 9 on-site interviews in one as well as 45 on-site interviews in another questionnaire testing

questionnaire in a particular business. In addition, an attempt was made to contact all the participants in the businesses of the second round. Because the role of other people was relatively minor, telephone interviews were conducted with them.

The observation was planned for a few of the ear- ly on-site visits to include any unexpected information in subsequent examinations and for a few businesses across the rest of the period. It was also intended in the new businesses to observe their original struggles in completing the QST questionnaire. If they had refused observation followed by an interview, a sole interview was requested. Observation was directed only to the person with the main role in completing the QST ques- tionnaire to avoid a serious disruption of the natural flow of the process.

4. Designing protocols for businesses

4.1. Steps in gaining cooperation

The designing of the contacts faced two key chal- lenges: they had to incorporate elements to gain the cooperation of and to motivate the person to reveal as much information as possible on the designated topic. Gaining cooperation relied prevalently on Dillman’s el- ements of the Tailored Design Method and the under- lying theory of social exchange [12]. It also considered the six psychological factors promoted by Cialdini: (1) reciprocation, (2) commitment and consistency, (3) so- cial validation or social proof, (4) authority, (5) scarcity, and (6) liking [7]. An important dilemma in this study was how to exploit the authority of the SORS to obtain participation and, at the same time, to convey indepen- dence from SORS to gain insights into the potentially sensitive adequacy of questionnaire completion.

The approach consisted of four steps. First, SORS sent an advance letter to alert the contact people in the

selected businesses. The appearance of the advance letters with the typical SORS layout (i.e., the SORS logo, stamp, and signature of the general director) tried to communicate the presence of a legitimate authority in the research, and the text clarified that the Faculty of Economics at the University of Ljubljana (FELU) would conduct the research independently of SORS. As Dillman suggests [12], the letters were personalized, concise, positively worded, and tried to trigger antici- pation and interest without too much detail. They pro- vided the respondents with essential information about the QST study. They stressed the confidentiality of data, the importance of participation, and the expect- ed benefit in the form of an easier survey response. They anticipated future activities and concluded with an acknowledgment.

The next step consisted of a short telephone conver- sation with the contact people who were to be the actual respondents of the QST questionnaire. The aim of the conversation was to obtain consent for cooperation and to acquire some information on the response process required for the efficient planning of data collection, particularly on the people involved in the process, the time spent filling in the questionnaire, and the timing of the next filling-in session. The protocol incorpo- rated several elements of the aforementioned theory of social exchange, such as the benefits for businesses in improving the questionnaire and in reducing a high re- sponse burden. In the introduction to the conversation, the interviewer immediately sought to establish trust by exposing two legitimate authorities, FELU and SORS. The former also aimed to invoke other exchange rela- tionships arising from its reputable status in higher eco- nomic and business education. If the respondents re- fused cooperation or exhibited uncertainty, several rea- sons were offered to convert or prevent the refusal. If this did not seem to work, the strategy was to postpone the final decision to the next telephone call. All the bur-

M. Bavdaž / Conducting research on the response process in business surveys 7

den of the future contacts remained with the researcher to avoid inconveniencing respondents.

In the middle of the period between the first tele- phone contact and the predicted filling in of the QST questionnaire, a second informative letter bearing only the FELU logo was sent to respondents by their pre- ferred means of communication. It briefly repeated the main points of the advance letter concerning the study, its usefulness, and confidentiality issues. It served as minutes of the telephone conversation quoting the agreed-on date of the next telephone contact. It ex- pressed a positive regard by providing the researcher’s contact information and an acknowledgment of the willingness to participate in the research, thus building up a relationship with respondents.

The second telephone call was made approximately one week before the probable date of completing the questionnaire, depending on the agreement with the respondent in the first telephone call. Its aim was to ask for final confirmation of the on-site visit or to determine another date to verify the status of filling in the QST questionnaire. An approximation of the length of the on-site visit had to be given as well. In the case of observation, the respondents knew the time needed to complete the questionnaire. Approximately a full hour to one-and-a-half hours was planned for the interview.

4.2. Protocols for the on-site visits

All contacts before the actual on-site visit had already provided a foundation for building a relationship with respondents. The on-site introductory verbal exchange had to create a relaxed atmosphere but also convey some key points. The respondents were thus asked about their business activity, which also provided some background information and helped in the evaluation of the subsequent answers (see [11]). The researcher also communicated the importance of the research, the independence from SORS, the confidentiality of the collected data, and a request to record the conversation.

Mentioning the independence from SORS and grant- ing confidentiality were not supposed to suffice for dealing with potential sensitivity issues. According to Lee, a distinction must be made between sensitive ques- tions and sensitive answers (as cited in [48]). The in- terview questions were not considered sensitive, while the same could not be claimed for the respondents’ answers. For instance, if respondents completed the questionnaire carelessly or intentionally ignored the in- structions, they might be reluctant to admit this. It was therefore necessary to maintain a neutral interviewer

approach with no overt reactions to the respondents’ answers and to focus on the defects of the questions and not of the person. The argument that many other businesses had problems with the QST questionnaire or some parts of it and that some questions could be problematic in certain activities had to be exposed at least once to relieve the respondents of any potential embarrassment, especially in the presence of other col- leagues.

After the introductory conversation, the focus moved to the last process of answering the QST questions be- fore the on-site visit. To gather information on the cognitive processes, the protocol included probes on question comprehension, data retrieval, judgment of the response adequacy, and communication of the re- sponse. It also adopted the characteristics of expan- sive or ethnographic interviewing to collect contextu- al information on the organization of the response and people involved, the role of authorities, and the infor- mation system. In the end, it addressed the statistical reporting itself. All the protocols tried to foresee and address as many possible reactions and developments as possible.

In those cases where an observation was conduct- ed, the protocol was interrupted after the introductory conversation. The observation was unstructured and thus did not follow any prescribed checklist. It was instead open to learning new insights. A respondent’s intermediate explanations and questions were kindly acknowledged but minimized. After completion of the questionnaire, the implementation of the protocol con- tinued, though it was adjusted for any gaps and findings from the observation.

4.3. Specifics for businesses new to the QST

The previously mentioned protocols were adapted to fit the businesses that participated in the QST for the first time. In businesses that were not obliged to com- plete the QST questionnaire, difficulties in obtaining consent for participation were expected, as the rewards were less likely to be realized in the short term.

The response process in these businesses was stud- ied using the Three-Step Test Interview (see [23]). The respondent was therefore observed during the self- completion of the questionnaire after being instructed to think aloud as much as possible. The interview af- terward first addressed the gaps in observational data and then moved to the same contextual issues as with people familiar with the QST.

8 M. Bavdaž / Conducting research on the response process in business surveys

5. Logistics of data collection

The logistics of data collection mainly concerned the efforts invested to recruit people involved in the re- sponse process in the business, other preparatory ac- tivities, and implementation. Previous experience with the involvement of business respondents in research activities was mixed. Several researchers pointed out the difficulties in gaining access to business respon- dents and in organizing events or activities with their participation [24,41,42]; yet others reported full coop- eration [32]. It thus seems that an adequate approach placing considerable attention to the respondents’ bur- den and motivation could tackle the problems at least to some degree. Unfortunately, detail about the recruit- ment step is often missing. In the QST study, all large businesses agreed to cooperation immediately, while much more effort was needed for the small and medium enterprises to give their consent.

The planning of field research had to take into ac- count the imposed survey deadlines on the one hand, and the business activities and priorities on the other hand. The goals were to be present for the observation of the response process and to minimize the elapsed time between the survey response and the on-site in- terview. In addition, businesses for observation had to be spread across the field period to include early and late respondents. For this reason, the businesses first contacted all received a request for observation until the planned spread was achieved.

The majority of respondents predicted the filling in relatively well. The impediments to minimizing the elapsed time included the respondent’s occupation with more important and urgent duties, and absence from work. In some rare cases, however, the request to con- duct the interview as soon as possible after the ques- tionnaire’s completion was misheard, forgotten, or ig- nored, thus creating a lag.

Preparations for every on-site visit were intensive and consisted of careful reviews of the available infor- mation collected through telephone calls and retrieved from commercial databases and the businesses’ home pages. It was left to the respondents to determine the place of the interview so that they would feel at ease. In fact, after observation, it was natural that the interview followed in the same room regardless of the presence of any others. Caution against the presence of others is suggested especially with sensitive questions and ques- tions that may produce a social desirability effect [45].

Detailed information on other people involved in the response process was collected during the on-site inter-

view with the main respondent. A serious drawback of this approach was the time that elapsed between their involvement in the response process and the interview, especially because it was generally a minor task, while an advantage was that the others were unaware of the research and therefore completed the task as usual. The latter was considered a priority in this research.

Although all the themes had to be covered in the in- terviews, there was some flexibility with the question sequence. The interview concentrated on those QST items that were actually filled in. A couple of hypo- thetical probes were subsequently included to tackle all response categories of an item that had been identified as potentially problematic. In fact, it turned out that respondents did not always understand all items and therefore decided to ignore them, which sometimes led to measurement errors.

The final number of businesses included in the study from the existing QST panel was 27, compared to the initial sample of 32 businesses. Only one business out of four that were new in the panel and became legally required to participate in the QST cooperated, as a result of inadequate timing of questionnaire delivery; and two businesses out of six not in the QST panel consented to a visit. Because none of the latter two engaged into the actual data retrieval, they were not included in the final analysis, though nothing contradicted other findings.

The main lesson learned from the implementation is that more insights into the first completion of the ques- tionnaire could be useful for understanding a respon- dent’s experience and successive response processes. This may be achieved by including a larger sample of businesses that will participate in the studied survey for the first time or by systematically searching for already participating businesses that will change their respon- dents for the next reporting period. It does not, how- ever, seem efficient to spend resources on businesses not legally required to report data, because of their low motivation to actually fill in the questionnaire.

6. Analysis of results

Qualitative research typically produces a wealth of data. This section presents methods of data analysis in the QST study and the sort of results they produced. It also discusses validity issues of findings based on the proposed qualitative approach.

M. Bavdaž / Conducting research on the response process in business surveys 9

6.1. Methods of data analysis

Data on the response process collected through the observations and interviews were recorded in the form of researcher’s notes and memos. The major part of the interviews was also tape recorded and transcribed. The transcription was not completely naturalized (see [37]). It departed from a verbatim transcription when the con- versation referred to unrelated issues (e.g., a telephone call), when the emphasis was on action and the words had to be put in context (e.g., retrieving data from the records), or when some noise prevented clear recording (e.g., in offices with more than one person).

The analysis of these data was time-consuming be- cause it required an intensive immersion in the data to transition from topics, questions, and data to answers or propositions (see [30]). In spite of several available methods, the central task of finding an approach resides with the researchers and their research questions [28]. Most important, the process has to use credible, reli- able, and replicable methods explicitly and systemati- cally [33].

The main themes of interview protocols served as a starting point for the process of data organization and later data reduction. The transcribed text was cut into segments. Relevant segments were attributed to the theme(s) they addressed and to the particular as- pect they illuminated within a theme. For instance, the study was interested in people involved in the response process, so all segments of text addressing this theme were gathered and then attributed to the person’s par- ticular role. The result was a comprehensive matrix that offered the possibility of examination by theme, by aspect, and by business. Further structuring of aspects within the themes was used to generate initial codes and crystallized through iterations. In subsequent rounds of text inspection, the coding became focused on and oriented toward the detection of patterns. At this point, the available material was reexamined to shed more light on the findings and to revise them accordingly: relevant parts of the interview transcriptions and notes from observations and interviews to relive the situation and get context data, and memos to check enlightening remarks and ideas. Some themes also required the ex- amination of transcriptions, notes, and memos from in- terviews with various experts. The patterns were pulled together into concepts and constructs with the help of various displays and with regard to previous study find- ings. Results were clustered around the sources of mea- surement errors and then integrated into a model of the survey response process, though the reverse order is also possible (see [19]).

6.2. Varieties of results

Qualitative research into the response process of the QST contributed to a further improvement in the ty- pology of sources of measurement errors in business surveys and the model of the survey response process (see [4]). In addition to these theoretical contributions, the research findings also have practical implications. They were an impetus for a reflection on the QST in- strument. Although the QST study did not produce quantitative indicators of measurement errors, it pro- vides argumentation that cannot be disregarded. This was, for instance, the case of a survey question re- questing a breakdown of wholesale turnover by buyers (retail, big intermediaries, manufacturing, wholesale, and final users). This breakdown was neither (readily) available in business records nor understood well, but that remained unnoticed in editing procedures. Data availability that accounting experts had already ques- tioned led to the use of only one category of buyers or to the repeated use of the same, initially estimat- ed, percentage breakdown by buyers, with percentages defined down to two decimal digits. All respondents who were asked to interpret the five categories of buy- ers finally exhibited some uncertainty, and those who reflected on the ideal data provided various solutions (e.g., classifying buyers by tax criteria, main activity, or institutional sectors; classifying every buyer’s intention regarding the merchandise). As a result, SORS took immediate measures after consulting the main user of information, the Department of National Accounts; the question was simplified to fewer categories of buyers and its periodicity reduced.

Other problematic questions, for instance, were breakdowns of wholesale turnover by 30 and retail turnover by 39 commodity groups. The breakdowns represented a challenge for all but the most special- ized businesses. Respondents had problems classify- ing their merchandise and matching their own groups with the requested ones. This resulted in unwanted practices in smaller and larger businesses: recourse to the option “other,” assignment of whole turnover to a single group, and an invariable percentage breakdown used quarter after quarter. SORS decided for further research before revising the instrument, as this problem did not have straightforward solutions.

Another practical contribution consisted of two sem- inars organized for people involved in business surveys at SORS. The seminars informed staff about the re- search findings and provided an opportunity to discuss the findings and problems, exchange ideas, and indicate possible solutions.

10 M. Bavdaž / Conducting research on the response process in business surveys

6.3. Validity issues

All steps and choices in designing the research, col- lecting the data, and conducting the data analysis were carefully documented, thereby creating the chain of ev- idence and enhancing the reliability of findings. When- ever possible, different sources of evidence addressed the same element of the response process (i.e., the same potential source of measurement errors) to enable trian- gulation. It must be kept in mind, however, that much of the data in qualitative research are context bound, which makes triangulation difficult or even problemat- ic [34].

In the on-site visits, nearly all respondents quickly overcame any initial tension or discomfort. The proto- cols stressed the importance of following the usual pro- cedure of the questionnaire’s completion and foresaw the sensitivity of the topic but dealt with it successful- ly. The evidence supporting this conclusion consists of several admissions during the second part of the in- terview and some corrections of previous “politically correct” answers or behaviors. For instance, a respon- dent who was observed while filling in the question- naire changed her strategy after my insistence that she carry out the task as usual: instead of retrieving some data, she provided estimates from memory based on previously reported data.

The effectiveness of on-site visits can be boosted by having one person conduct discussion and another per- son observe nonverbal behavior and take notes. How- ever, in such a case, it may be more difficult to create a relaxing atmosphere to deal with sensitivity issues.

The chief drawback of interviewing concerned the time that elapsed between the filling in of the QST ques- tionnaire and the on-site visit. This time had to be mini- mized so that respondents forgot the minimum of infor- mation. The significant efforts invested in scheduling the on-visits paid off, as half of the businesses worked on the questionnaire on the day of the on-site visit, though they were at different stages of completing the questionnaire. Longer time gaps could have deterio- rated the recall, but three factors worked against this: the advance announcement of the impending on-site visit and the recurrence of the response process both enhanced the encoding of associated memories, and the accompanying working papers figured as excellent memory cues.

An important validity issue concerns the extent that the study findings can be generalized, given the specifics of the present research. The first impedi- ment to generalization was the sample, which was too

small and nonrepresentative to concede a statistical in- ference. The second impediment were the specifics of the QST with regard to its mode, recurrence, pe- riodicity, questionnaire length and complexity, topics covered, mandatory character, and so on. However, the present research relies on analytical generalization (see Stake, as cited in [28]). It is therefore necessary to examine the specifics of selected businesses and the selected survey and judge them against businesses with other characteristics.

Participating businesses varied from microenterpris- es engaged in one narrowly defined trade activity to the largest businesses with a mixture of trade and nontrade business activities. The businesses also differed with regard to legal personality, origin of capital, geograph- ical location, group participation, involvement in inter- national activities, and organizational culture. In addi- tion, the data were collected for two reporting periods (i.e., the last and the first quarter of the calendar year) to incorporate differential burdens imposed on busi- nesses from regulatory authorities and management. Some respondents also worked in accounting firms, so they likely behaved similarly when completing other types of questionnaires for other types of businesses. Therefore, the selected businesses may figure as rep- resentatives of many nonselected ones. Nevertheless, some caution is necessary when applying the findings to “pure” nontrade businesses and across borders to environments with different institutional contexts and business environments. Furthermore, the research does not provide a conclusion about the few businesses that refused participation in the study because they were overworked.

The QST study is applicable to self-administered sur- veys, though those conducted on the Web or in any oth- er electronic version must consider the impact of tech- nology on the response process. The analogies with interviewer-administered surveys are, however, limit- ed. There was also more focus on the recurring re- sponse process than on the first- or only-time response process. However, quarterly periodicity may be per- fect for an analysis of recurrence, as its effects become strong enough to be captured compared to annual or less frequent surveys and weak enough to be traced before becoming too fast a routine.

The actual length and complexity of the QST ques- tionnaire depended on the multiplicity of business ac- tivities, so the completion of the questionnaire could range from very simple and quick to demanding and time consuming. The QST covered several topics that determined the occupation and number of people in-

M. Bavdaž / Conducting research on the response process in business surveys 11

volved in the response process. In some cases, one respondent providing all the necessary data handled all questions; other cases required the coordination of dif- ferent data providers. The generalization may thus be made to both situations. Although the QST mainly en- gaged accounting staff, it was not limited to them. It also involved people from financial, advertising, and personnel departments, as well as a few technicians. The study may thus provide useful insights for busi- ness surveys involving people with economic and other backgrounds; it may fail to do so only with very specific topics.

This research addressed only marginally the effects of the QST’s mandatory character and governmental sponsorship. The findings may still apply to nonmanda- tory and nongovernmental business surveys, but con- siderable caution is necessary given the expected differ- ential perceptions of the survey request and disputable motivation for participation and accurate completion. Business surveys also represent a special type of orga- nizational and establishment survey. The range of sim- ilarities may provide solid foundations for the applica- tion of the findings to other types of organizational and establishment surveys. The differences, however, need to be taken into consideration.

7. Recommendations for survey work

This section gathers some general recommendations for how to conduct qualitative research on the response process in business surveys.

– Use multiple sources of evidence to shed light on the response process from different perspectives, especially from a description given by the people involved in the response process in the business and statistical organization, from an observation of the response process, and from several experts with different experiences.

A well-considered combination of experts may pro- vide a satisfactory examination of the response process in business surveys and may be relatively cheap and easy to achieve without burdening businesses. Subject- matter experts may be indispensable in highlighting substantive and terminological issues; questionnaire design experts may be essential in evaluating if the survey instrument communicates its contents well; and questionnaire administration experts may be helpful for providing feedback from the businesses and other expe- riences with them. A review of completed forms may suggest where and how measurement errors occur.

Nevertheless, only fieldwork seems to enable a com- prehensive insight into the response process from the business perspective. In this regard, observation has a limited potential if it is not supported by audio and video recording as well as electronic tracking, given the intensive interactions of business participants with various devices. However, the comprehensive tracking of an employee’s activities is a sensitive issue because of the considerable intrusion into that person’s privacy and business confidentiality. As a result, interviewing still is the most powerful method for collecting data on the actual response process. If the respondent is per- forming the task of answering survey questions for the first time, the thinking-aloud technique may be most ef- fective. If the respondent is familiar with the task, only retrospective probing seems appropriate. In both cases, the addition of ethnographic interviewing may be valu- able for providing more insights into comprehension and the use of economic concepts and conditions from the business environment.

Observation should not be totally discarded even without technological support and despite a probable modest ratio between time consumption and the amount of data collected, as it may provide unexpected infor- mation. It seems especially important in combination with interviewing for at least two reasons, both of which enhance the validity of collected data. First, it min- imizes the elapsed time between the actual response process and reporting about it. Second, it relativizes the information collected during the subsequent inter- view because it enables a comparison of the intervie- wee’s and researcher’s perceptions of the same issue, such as the invested effort and time, and the conduct of activities.

– Select businesses of different sizes but check vari- ability in other characteristics likely to influence the response process. Select people new to the sur- vey task and those familiar with it. Focus on people who provide data with the particular purpose of answering survey questions, but do not completely exclude other business participants from the study.

Size of a business is considered the main determinant of the survey response process, but other characteristics may also be relevant, particularly kinds of economic activity, outsourcing of the survey task, timing of the survey task with respect to the deadline and other legal requirements, legal personality, origin of capital, geo- graphical location, group participation, involvement in international activities, organizational culture, and so on. Response process also changes with recurrence,

12 M. Bavdaž / Conducting research on the response process in business surveys

which makes it important to study businesses that par- ticipate in a survey for the first time and new respon- dents in previously participating businesses in addi- tion to recurrent respondents. Although respondents are those who eventually carry out the survey task, the inclusion of a few other business participants such as authorities may provide new insights.

– Gain cooperation of business participants by ex- ploiting each of several contacts with them to build a relationship and justify the research.

Research involving businesses requires that sufficient time is allotted to gain the cooperation of business par- ticipants. More effort may be necessary for small and medium-sized enterprises. The differential motivation for cooperation can be explained by the social responsi- bility of the larger businesses and the limited resources of the smaller ones. Also, it seems beneficial to expose the rewards of participation for the business. Because participation in research necessarily represents a cost to businesses, it is crucial that their burden is reduced to the minimum whenever possible.

– Include businesses responding early and late but always minimize the elapsed time between the sur- vey response process in the business and the data collection to ensure validity of the collected data.

Once consent is obtained, plan for the timing of on- site visits. On-site visits need to be scheduled when the process will be going on or right after the task has been completed. Because business participants may not comprehend the importance of such scheduling for the validity of findings, the researchers should emphasize it. It is also necessary to include businesses that respond at different times, as this may be related to the source of measurement errors.

– Design protocols for on-site visits thoroughly and prepare to deal with sensitivity and confidentiality issues.

The available time on-site must be used wisely to ac- quire comprehensive information on the response pro- cess and relevant context. Obtaining a candid insight into the response process may be challenging if the questionnaire was not completed (entirely) in line with instructions and if sanctions apply. It is therefore nec- essary to relieve the respondents of any potential em- barrassment and sanctions by relativizing problematic practices. Conducting research in cooperation with an independent and trustworthy institution may further re- duce this problem. Confidentiality issues may be ad-

dressed by focusing exclusively on the processes pro- ducing survey data and not on the data themselves. However, some information on the relevant context may still be classified, so granting confidentiality seems in- dispensable.

– Document all steps and decisions made in research and allow sufficient time for analysis that can be conducted by sources of measurement errors and by response processes.

The engagement in the preparation and implementation of fieldwork should not compromise the documenta- tion practices and data analysis. Good documentation allows the reconstruction of all research activities and the durability of the collected data to exploit the usual richness of qualitative data. Audio recording appears to be indispensable for ensuring high-quality transcrip- tions, though notes and memos are important as well. Analysis of collected data by survey questions seems most natural, but it may prevent absorption of the big picture and detection of general patterns that could be added to the body of knowledge on measurement errors in business surveys. It is therefore recommended to analyze the data by all sources of measurement errors and by separate response processes.

– Learn and train. Disseminate findings.

The quality of qualitative research highly depends on the ability of researchers. It seems vital that researchers who study the response process in business surveys have enough methodological and domain knowledge as well as communication skills. New insights can be particularly beneficial and effective if incorporated in training staff at the survey organization such as ques- tionnaire administration experts and if conveyed to all other relevant parties.

8. Conclusion

To summarize, the evaluation of measurement er- rors starts with the identification of their sources in the survey response process. To obtain an in-depth un- derstanding of the response process, this article advo- cates the use of qualitative research methods. Howev- er, such methods have strengths and weaknesses, and researchers must be aware that the business environ- ment and specifics of business surveys usually impose additional methodological and logistical challenges on the research design and implementation. In the fu- ture, more research on methods and other procedures is

M. Bavdaž / Conducting research on the response process in business surveys 13

needed to determine their strengths and shortcomings as well as their cost-effectiveness for studying the re- sponse process of different types of business surveys in various business environments.

Acknowledgments

I thank the staff of the Statistical Office of the Re- public of Slovenia for cooperation, as well as Lars Ly- berg, Lea Bregar, and Jaak Billiet for their guidance and support in this research.

References

[1] M.V. Angrosino, Ethnography, pedagogy, and the prospects for a progressive political agenda, in: The SAGE handbook of qualitative research, (3rd ed.), N.K. Denzin and Y.S. Lincoln, eds, Sage Publications, Thousand Oaks, 2005, pp. 729–745.

[2] C. Babyak, A. Gower, L. Gendron, J. Mulvihill and R.-A. Zaroski, Testing of the questionnaires for Statistics Canada’s Unified Enterprise Surveys, Proceedings of the Second Inter- national Conference on Establishment Surveys ICES-II, Amer- ican Statistical Association, Alexandria, 2000, pp. 317–326.

[3] I. Baszanger and N. Dodier, Ethnography: Relating the part to the whole, in: Qualitative research: Theory, method and prac- tice, (2nd ed.), D. Silverman, ed., Sage Publications, London; Thousand Oaks, 2004, pp. 9–34.

[4] M. Bavdaž, Measurement errors and the response process in business surveys, Ph.D. Dissertation, University of Ljubljana, 2007.

[5] P. Beatty, The dynamics of cognitive interviewing, in: Meth- ods for testing and evaluating survey questionnaires, S. Press- er, J.M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin, J. Mar- tin et al., eds., Wiley-Interscience, Hoboken, 2004, pp. 45–66.

[6] P. Beatty, S. Schechter and K. Whitaker, Variation in cognitive interviewer behavior – extent and consequences, Proceedings of the Survey Research Methods Section, 1997, Retrieved February 19, 2004 from http://www.amstat. org/sections/srms/Proceedings/papers/ 1997 183.pdf.

[7] P.P. Biemer and L.E. Lyberg, Introduction to Survey Quality, Wiley-Interscience, Hoboken, 2003.

[8] K. Charmaz, Grounded theory in the 21st century: Appli- cations for advancing social justice studies, in: The SAGE handbook of qualitative research, (3rd ed.), N.K. Denzin and Y.S. Lincoln, eds, Sage Publications, Thousand Oaks, 2005, pp. 507–535.

[9] D. Collins, Pretesting survey instruments: An overview of cognitive methods, Quality of Life Research 12 (2003), 229– 238.

[10] F.G. Conrad and J. Blair, Data quality in cognitive interviews: The case of verbal reports, in: Methods for Testing and Eval- uating Survey Questionnaires, S. Presser, J.M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin and J. Martin, eds, Wiley- Interscience, Hoboken, 2004, pp. 67–87.

[11] C. Cosenza, Not your grandparent’s cognitive testing: Ex- ploring innovative methods in cognitive evaluation of ques- tions, 2003 Joint Statistical Meetings - Section on Govern- ment Statistics [CD-ROM], American Statistical Association, Alexandria, 2003, pp. 1058–1063.

[12] D.A. Dillman, Mail and Internet surveys: The tailored design method, (2nd ed.), John Wiley & Sons, New York, 2007.

[13] W.S. Edwards and D. Cantor, Toward a response model in es- tablishment surveys, in: Measurement Errors in Surveys, P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz and S. Sudman, eds, Wiley-Interscience, New York, 1991, pp. 211– 233.

[14] J. Eldridge, J. Martin and A. White, The use of cognitive meth- ods to improve establishment surveys in Britain, Proceedings of the Second International Conference on Establishment Sur- veys ICES-II, American Statistical Association, Alexandria, 2000, pp. 307–316.

[15] B.H. Forsyth and J.T. Lessler, Cognitive laboratory methods: A taxonomy, in: Measurement Errors in Surveys, P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz and S. Sudman, eds, Wiley-Interscience, New York, 1991, pp. 393–418.

[16] B.H. Forsyth, K. Levin and S.K. Fisher, Test of an appraisal method for establishment survey questionnaires, Proceedings of the Survey Research Methods Section, 1999, Retrieved February 18, 2004 from http://www.amstat.org/sections/srms/ Proceedings/papers/1999 021.pdf.

[17] B.H. Forsyth, E.S. Weiss and R. Miller Anderson, A compar- ison of appraisal and cognitive interview methods for testing organizational survey questionnaires, International Confer- ence on Questionnaire Development, Evaluation and Testing Methods QDET, 2002, Retrieved April 29, 2005, from http:// www.jpsm.umd.edu/qdet/final pdf papers/round%20two/ Forsyth.pdf.

[18] E.R. Gerber, The view from anthropology: Ethnography and the cognitive interview, in: Cognition and survey research, M.G. Sirken, D.J. Herrmann, S. Schechter, N. Schwarz, J.M. Tanur and R. Tourangeau, eds, Wiley-Interscience, New York, 1999, pp. 217–234.

[19] D. Giesen, The response process model as a tool for evaluat- ing business surveys, Proceedings of the Third International Conference on Establishment Surveys ICES-III [CD-ROM], American Statistical Association, Alexandria, 2007, pp. 327– 337.

[20] D. Giesen and T. Hak, The response process model in business surveys: Lessons learned by using a multi- method approach, 2005 FCSM Conference Papers, 2005. Re- trieved April 13, 2006 from http://www.fcsm.gov/05papers/ Giesen Hak IXB.pdf.

[21] A.C. Graesser, Z. Cai, M.M. Louwerse and F. Daniel, Ques- tion understanding aid (QUAID), Public Opinion Quarterly 70 (2006), 3–22.

[22] R.M. Groves, F.J. Fowler, Jr., M.P. Couper, J.M. Lepkowski, E. Singer and R. Tourangeau, Survey methodology, Wiley- Interscience, Hoboken, 2004.

[23] T. Hak, K. van der Veer and R. Ommundsen, An application of the Three-Step Test-Interview (TSTI): A validation study of the Dutch and Norwegian versions of the ‘Illegal Aliens Scale’, International Journal of Social Research Methodology 9 (2006), 215–227.

[24] T. Hak, D.K. Willimack and A.E. Anderson, Response pro- cess and burden in establishment surveys, 2003 Joint Statisti- cal Meetings - Section on Government Statistics [CD-ROM], American Statistical Association, Alexandria, 2003, pp. 1724– 1730.

[25] T. Holt and T. Jones, Quality work and conflicting quality ob- jectives, 1998. Retrieved March 25, 2004, from http://www. statistics.gov.uk/methods quality/downloads/QualityWork. pdf.

14 M. Bavdaž / Conducting research on the response process in business surveys

[26] C.R. Jenkins and D.A. Dillman, Towards a theory of self- administered questionnaire design, in: Survey measurement and process quality, L.E. Lyberg, P.P. Biemer, M. Collins, E. de Leeuw, C.S. Dippo, N. Schwarz and D. Trewin, eds, Wiley-Interscience, New York, 1997, pp. 165–196.

[27] G. Kamberelis and G. Dimitriadis, Focus groups: Strategic articulations of pedagogy, politics, and inquiry, in: The SAGE Handbook of Qualitative Research, (3rd ed.), N.K. Denzin and Y.S. Lincoln, eds, Sage Publications, Thousand Oaks, 2005 , pp. 887–907.

[28] S. Kvale, InterViews: An Introduction to Qualitative Research Interviewing, Sage Publications, Thousand Oaks, 1996.

[29] J.T. Lessler and B.H. Forsyth, A coding system for appraising questionnaires, in: Answering Questions: Methodology for determining Cognitive and Communicative Processes in Sur- vey Research, N. Schwarz and S. Sudman, eds, Jossey-Bass Publishers, San Francisco, 1996, pp. 259–318.

[30] J. Lofland and L.H. Lofland, Analyzing Social Settings: A Guide to Qualitative Observation aAnd Analysis, (3rd ed.), Wadsworth, Belmont, 1995.

[31] L. Lyberg and E. Elvers, Survey quality issues during the last 50 years. Some observations, 2003 Joint Statistical Meetings – Section on Survey Research Methods [CD-ROM], American Statistical Association, Alexandria, 2003, pp. 2597-2604.

[32] A.W. Martin, R.K. Adler and N.L. Leach, Assessing the im- pact of the redesign of the Manufacturing energy consump- tion survey, Proceedings of the Second International Confer- ence on Establishment Surveys, ICES-II, American Statistical Association, Alexandria, 2000, pp. 1488–1492.

[33] M.B. Miles and A.M. Huberman, Qualitative Data Analysis: An Expanded Sourcebook, Sage Publications, Inc., Thousand Oaks, 1994.

[34] G. Miller and K.J. Fox, Building bridges: The possibility of analytic dialogue between ethnography, conversation analy- sis and Foucault, in: Qualitative Research: Theory, Method and Practice, (2nd ed.), D. Silverman, ed., Sage Publications, London; Thousand Oaks, 2004, pp. 35–55.

[35] E.M. O’Brien, Respondent role as a factor in establishment survey response, Proceedings of the Second International Conference on Establishment Surveys, ICES-II, American Sta- tistical Association, Alexandria, 2000, pp. 1462–1467.

[36] E. O’Brien, S.K. Fisher, K. Goldenberg and R. Rosen, Ap- plication of cognitive methods to an establishment survey: A demonstration using the Current Employment Statistics Sur- vey, Proceedings of the Survey Research Methods Section, 2001. Retrieved January 2, 2002 from http://www.amstat. org/sections/srms/Proceedings/y2001/Proceed/00031.pdf.

[37] D.G. Oliver, J.M. Serovich and T.L. Mason, Constraints and opportunities with interview transcription: Towards reflection in qualitative research, Social Forces 84 (2005), 1273–1289.

[38] C. Ramirez, Strategies for subject matter expert review in questionnaire design, International Conference on Question- naire Development, Evaluation and Testing Methods QDET, 2002, Retrieved August 6, 2006, from http://www.jpsm.umd.

edu/qdet/final pdf papers/ramirez.pdf. [39] C.D. Redline and C P. Lankford, Eye-movement anal-

ysis: A new tool for evaluating the design of visual- ly administered instruments (paper and web), Proceedings of the Survey Research Methods Section, 2001. Retrieved August 3, 2006 from http://www.amstat.org/sections/srms/ Proceedings/y2001/Proceed/00248.pdf.

[40] O. Rowlands, J. Eldridge and S. Williams, Expert review fol- lowed by interviews with editing staff – effective first steps in the testing process for business surveys, International Confer- ence on Questionnaire Development, Evaluation and Testing Methods QDET, 2002. Retrieved August 6, 2006, from http: //www.jpsm.umd.edu/qdet/final pdf papers/rowlands.pdf.

[41] G. Snijkers, Cognitive Laboratory Experiences: On Pre- Testing Computerised Questionnaires aAnd Data Quality, Ph.D. Dissertation, Universiteit Utrecht, 2002.

[42] K. Stettler, D.K. Willimack and A.E. Anderson, Adapt- ing cognitive interviewing methodologies to compensate for unique characteristics of establishments, Proceedings of the Survey Research Methods Section, 2001. Re- trieved January 2, 2002 from http://www.amstat.org/sections/ srms/Proceedings/y2001/Proceed/00011.pdf.

[43] S. Sudman, D.K. Willimack, E. Nichols and T.L. Mesenbourg, Exploratory research at the U.S. Census Bureau on the sur- vey response process in large companies, Proceedings of the Second International Conference on Establishment Surveys, ICES-II, American Statistical Association, Alexandria, 2000, pp. 327–337.

[44] D. Tomaskovic-Devey, J. Leiter and S. Thompson, Organiza- tional survey nonresponse, Administrative Science Quarterly 39 (1994), 439–457.

[45] S. Tu, The presence of others and survey response, Pro- ceedings of the Survey Research Methods Section, 2000. Re- trieved August 5, 2006, from http://www.amstat.org/sections/ srms/Proceedings/papers/2000 157.pdf.

[46] D.K. Willimack, Business respondents’ perspectives on al- ternative employment arrangements and implications for em- ployment statistics, 2003 Joint Statistical Meetings – Section on Government Statistics, American Statistical Association, Alexandria, 2003, pp. 4559–4570.

[47] D.K. Willimack, L/E. Lyberg, J. Martin, L. Japec and P. Whitridge, Evolution and adaptation of questionnaire devel- opment, evaluation, and testing methods for establishment sur- veys, in: Methods for testing and evaluating survey question- naires, S. Presser, J.M. Rothgeb, M.P. Couper, J.T. Lessler, E. Martin and J. Martin, eds, Wiley-Interscience, Hoboken, 2004, pp. 385–407.

[48] G.B. Willis, Cognitive Interviewing: A tool for Improving Questionnaire Design, Sage Publications, Thousand Oaks, 2005.

[49] G.B. Willis, P. Royston and D. Bercini, The use of verbal re- port methods in the development and testing of survey ques- tionnaires, Applied Cognitive Psychology 5 (1991), 251–267.

emailed to multiple sites or posted to a listserv without the copyright holder's express written permission.

However, users may print, download, or email articles for individual use.