RCH 5301 VI PP

RCH 5301, Research Design and Methods 1

Course Learning Outcomes for Unit III At the end of this unit, you should be able to:

4. Apply research methods within a research design. 4.2 Develop a survey instrument for a research problem(s).

Required Unit Resources Chapter 5: The Introduction (ULO 4.2) Chapter 6: The Purpose Statement (ULO 4.2) Chapter 8: Quantitative Methods (ULO 4.2) Read the following sections from this chapter:

• Components of a Survey Study Method Plan

• The Survey Design o The Purpose o Rationale for Using the Survey Method o Type of Survey Method o Specify the Form of Data Collection o The Population and Sample

▪ The Population ▪ Sampling Design ▪ Type of Sampling ▪ Population Stratification ▪ Sample Size Determination ▪ Power Analysis

o Instrumentation ▪ The Survey Instruments used to Collect Data ▪ Instruments and Measures ▪ Validity of Scores Using the Instrument ▪ Reliability of Scores on the Instrument ▪ Inter-Rater Reliability ▪ Sample Items ▪ Content of an Instrument ▪ Pilot Testing ▪ Administering the Survey

o Variables in the Study Article: What You Should Know About Using Surveys (ULO 4.2) This article provides more information about implementing surveys. (3 pages) Video: Samples and Surveys: Against All Odds—Inside Statistics (ULO 4.2) Watch the following video segments to learn more about surveys. The transcript for this video can be found by clicking on “Transcript” in the gray bar to the right of the video in the Films on Demand database.

Survey Center (Segment 2) (2 minutes) Survey Research (Segment 3) (2 minutes)

UNIT III STUDY GUIDE

Primary Data Collection and Surveys

RCH 5301, Research Design and Methods 2

UNIT x STUDY GUIDE

Title

Unit Lesson Lesson: Everything You Wanted to Know About Surveying but Were Afraid to Ask (ULO 4.2)

Surveying Introduction Surveying is a systematic process of collecting data. Surveying falls into two general categories; census surveys and sample surveys. A census survey includes collecting data from every individual case in the population of interest. A sample survey collects data from a sampling frame, which is a subset or list of all population cases from which a sample is drawn. The ultimate purpose of surveying and sampling, as was discussed in Unit II, is to use the sample data to make inferences about a population of interest, which can include humans, objects, and events. Surveying is especially effective if the researcher has a good idea of the effect they expect to find since they can formulate survey items to test those expectations or hypotheses. Surveys often capture data related to participant attitudes, beliefs, perceptions, and opinions and are particularly insightful for disciplines such as psychology, marketing, human resources, health care administration, occupational safety and health, and criminal justice.

Surveying and the Sampling Frame The sampling frame should ideally represent the population, although it is rarely a perfect exemplar. For example, assume the director of student affairs at a large university is interested in obtaining feedback from the members of the current freshman class about their experience in acclimating to the university in the last fall term. The director decides to use a sample survey distributed in the following spring term. The sampling frame includes 200 freshmen housed in the largest dorm on the main university campus since directory data is readily available for these students. While access to the freshman dorm residents may be convenient for the director, it is not the best representation of the population of all freshmen enrolled in the university in the prior fall term. The sampling frame suffers from the following problems:

it excludes freshmen living in off-campus dorms and housing, who commute, it excludes freshmen at satellite campuses, it excludes freshmen who transferred or withdrew from the university during or after the fall term, it includes freshmen who enrolled in the spring term, and it does not factor in the possibility that the directory information may be incorrect.

One glaring problem with this sampling frame is that many of the freshmen in the population are being excluded. Furthermore, the excluded students are likely to have been some of the most dissatisfied and disaffected among the freshman class since they opted for non-campus living arrangements and/or unenrolled from the university altogether. As a result, the director may fail to capture the very data she is seeking to make informed decisions to improve the student acclimation experience.

Interesting and Potentially Detrimental Source of Bias Rosenthall and Rosnow (1975) discovered that volunteers and non-volunteers possess unique characteristics relative to their respective groups. This is a cautionary tale about how relying too heavily on one group or the other could significantly bias responses and, therefore, threaten the ability to generalize survey results to a population. According to Rosenthall and Rosnow (1975), volunteers tend to be better educated than non- volunteers, tend to have higher social-class status than non-volunteers, tend to be higher in need for social approval than non-volunteers, tend to be less authoritarian than non-volunteers, tend to be from smaller towns than non-volunteers, tend to be more self-disclosing than non-volunteers, tend to be more maladjusted than non-volunteers, are more likely to be married than non-volunteers, are more likely to be firstborns than non-volunteers, and tend to be more anxious than non-volunteers. Although the impact of a sampling frame of volunteers versus non-volunteers would depend on the nature of the study, it is clear that results could be disastrous. For example, assume a criminal justice researcher is interested in studying the emotional intelligence of recidivists convicted of felonies in U.S. courts. If the sampling frame includes only volunteers, the results are likely to be erroneously skewed toward higher emotional intelligence and not truly representative of the population of interest (Borg & Gall, 1979).

RCH 5301, Research Design and Methods 3

UNIT x STUDY GUIDE

Title

Benefits and Costs of Surveying In a perfect world, a census would be used to gather data about a population. Unfortunately, time, cost, and accuracy are important considerations for the researcher, and rarely are these three considerations not limiting factors to performing a census. Opting for a sample survey reduces the effort required for collecting and analyzing data versus attempting a census. Spending less time collecting and analyzing data translates into time and cost savings, which is a competitive advantage since time is of the essence when making decisions in dynamic and fluid environments. Using sample surveys can speed up the decision-making process markedly, especially when the populations are large and inaccessible. Since questionnaires are standardized using closed-ended questions, it is not necessary for costly training of interviewers. In some cases, standardization through questionnaires can eliminate the need for structured interviews (and associated travel costs) entirely through rapid and simultaneous distribution of questionnaires to large groups of people. Self-completion questionnaires improve accuracy by eliminating bias of both the interviewer and interviewee. For example, researchers do not have to be concerned with interviewers going off script, explaining, or changing the order of questions, and they do not have to worry about interviewee feelings toward the interviewer biasing responses. Finally, the anonymity and confidentiality that can be arranged through self-completion questionnaires is believed to contribute to more honest responses from participants since they do not feel the threat of being judged or retaliated against (Cooper & Schindler, 2014). Regardless of the advantages of using a sample survey, such as lower cost per respondent, there are other elements of surveying that should be contemplated. A primary risk is that the survey data may not accurately represent the population. Notwithstanding the danger of making decisions based upon invalid and unreliable survey results, this would be the worst-case scenario since much effort is generally needed for even small surveys. While internet technology has greatly lowered the time required for surveying, there are other costs that must be recognized. Surveying requires planning the project, designing, and testing the survey instrument, coordinating permissions, accessing participants, collecting and analyzing the data, and meeting required deadlines (Cooper & Schindler, 2014). Other concerns of self-completion questionnaires include misinterpretation or misunderstanding of survey instructions or questions and threats to validity by respondents intentionally misleading, missing data, and low response rates. Low response rates can be especially problematic in random samples. Although most novice researchers believe low response rates are mainly detrimental to achieving the required sample size, this is easily overcome by distributing more surveys to the sampling frame. The greatest threat presented by low response rate is systematic error. Since those failing to complete surveys may represent a special group in the sample, results may be biased. For example, employees who harbor distrust of their employer and fear retaliation are unlikely to participate in a human resources survey attempting to measure employee morale, satisfaction, and motivation. Similarly, demoralized, dissatisfied, and unmotivated employees fearful of retaliation may respond dishonestly by telling the employers what they want to hear regarding these concepts. In either case, results will not accurately portray the population of employees (Ross et al., 2002). Some problems of surveying are less obvious than poor question design or low response rates. Some researchers find themselves so focused on the mechanical aspects of surveying, such as the sampling frame, sample size, and instrument design, that they neglect the impact they may have on the organization simply by virtue of questioning employees. Sometimes, when using samples in organizations, those employees who were not randomly selected feel neglected. They may feel that they did not receive a questionnaire because their input is not valued. Conversely, employees randomly selected may feel targeted. Still, others may view the researcher as an agent of management who is tasked with finding a way to undermine employees. All these things must be considered as a part of the survey plan, and appropriate messaging should be developed to allay fears and misconceptions. If the population is small enough, the cost of using a census should be weighed against the benefit of not upsetting the organizational social system by using a sample (Fowler, 2009).

The Quantitative Data Collection Method The choice of data collection method is ultimately guided by the statement of problems, research questions, and hypotheses that have been developed. If the survey is the optimal data collection method, it must be designed so that the data it produces is aligned with the hypotheses that have been formulated. While hypotheses may be developed to test several different concepts, the survey items themselves are written to

RCH 5301, Research Design and Methods 4

UNIT x STUDY GUIDE

Title

measure the concepts. For example, there may be three hypotheses to test the relationship between the concepts of employee job satisfaction, motivation, and morale and the dependent variable of productivity. The questionnaire may include five items that measure job satisfaction, seven items that measure motivation, and six items that measure morale. Best practices in research suggest that each concept in a survey should be measured by a minimum of five to seven questions. The more complex the concept, the more questions that are generally needed to adequately measure the concept (Zikmund et al., 2013). Similarly, the survey must be designed with advanced knowledge of the inferential statistics to be used to test the hypotheses. For example, regression analysis works best with continuous data or data that is measured on an interval or ratio scale, so questions should be written appropriately to capture this type of data. The survey instrument must also align with the sampling design. If participants in the sampling frame are English speakers who are likely to have an 8th-grade education, questions must be written appropriate to that reading level or, preferably, lower. Survey questions must also be written so that they are closed-ended and conducive to data reduction, or the process of taking the voluminous raw data and reducing it to accurate and manageable formats. This is especially important if manual paper surveys or scanner sheets are used (Zikmund et al., 2013).

Survey Instrument Design The survey instrument is often referred to as a self-completion questionnaire, self-administered questionnaire, survey, or questionnaire. Quantitative research strategies use self-administered questionnaires extensively in academic and private sector research and increasingly more often today given the ubiquity of global internet adoption. Bell et al. (2022) offer an excellent description of the different types of questionnaires. In customary usage, when people speak of questionnaires (used in either self-completion or structured interviews), they are referring to those instruments using closed-ended questions. Questionnaires using open-ended questions begin to fall into the domain of interviewing, which is a qualitative research method. Responses to open- ended questions are often difficult, if not impossible, to quantify and standardize on a scale, and are, therefore, left open to researcher interpretation. This clearly falls under the interpretivist research tradition. Parts of the Questionnaire A self-completion questionnaire should be comprised of three parts: 1) the cover, 2) the instructions, and 3) the survey items. The cover should explain the purpose of the study, identify the sponsor of the study, describe why the respondent has been chosen to participate, encourage truthful and timely responses, and motivate the participant. Additionally, the expected timeframe for completion should be included and, if applicable, instructions for returning the survey with a contact name, email or physical address, and telephone number should be provided. If participant names will be collected, the cover should also include a confidentiality statement. The cover letter should thank the participants for their valuable time and input (Ross et al., 2002). The instructions should clearly explain how to complete the questionnaire in familiar language. The importance of clear instructions cannot be understated since this is the only information participants will have in the absence of the researcher. Researchers should anticipate any potential questions, confusion, or ambiguity the instructions may create. It is recommended to avoid or, at the very least, define any obscure terms. It is also a best practice to describe the response scale that will be used and include an example survey item. Notice the use of the word item. This is because questionnaires, despite the name, do not always contain questions. Often, the questionnaire items are statements that participants are asked to rate their intensity of agreement (Ross et al., 2002). The following depicts how a researcher might present the scale and provide an example item in the instructions.

RCH 5301, Research Design and Methods 5

UNIT x STUDY GUIDE

Title

The following scale will be used to gauge your responses to the questions. This scale will be repeated after each survey item. You will be asked to select one, and only one, choice for each survey item.

• Strongly agree

• Agree

• Neither agree nor disagree

• Disagree

• Strongly disagree Example survey item:

The goals of the organization are clear to me. ___ Strongly agree ___ Agree ___ Neither agree nor disagree ___ Disagree ___ Strongly disagree

Questions should be clear and non-ambiguous, use non-inflammatory language, and have face validity or appear to participants that they are consistent with the purpose of the study. This requires attention and finesse when writing survey questions, which are usually demographic questions (e.g., age, gender, ethnicity, job classification) and intensity questions (e.g., rate intensity of your attitudes, beliefs, opinions) (Seashore et al., 1983). Job Satisfaction Survey Example The following is an example of a job satisfaction survey. Note that items are included for various dimensions of the concept of job satisfaction, including pay satisfaction, promotion satisfaction, supervision satisfaction, and co-worker satisfaction. The questions with (R) following the item indicate that reversed scoring is required. This is intentional to determine if responses are consistent or if participants are simply randomly checking boxes. When the data is cleansed for analysis, reversed scored items are adjusted. For example, if a respondent selects 1 for “employee wages are commensurate with the work they perform” and selects a 6 for “raises are rarely given,” the 6 will be reversed to a 1 for data analysis so all responses are in the same direction. Responses are obtained on a 6-point Likert scale where 1 = disagree very much, 2 = disagree moderately, 3 = disagree slightly, 4 = agree slightly, 5 = agree moderately, 6 = agree very much.

RCH 5301, Research Design and Methods 6

UNIT x STUDY GUIDE

Title

RCH 5301, Research Design and Methods 7

UNIT x STUDY GUIDE

Title

RCH 5301, Research Design and Methods 8

UNIT x STUDY GUIDE

Title

(Adapted from Fields, 2002)

Advice for Survey Construction

Researchers who have designed and implemented many surveys often remark that problems that occur after launch could have almost always been prevented if more care would have been taken during construction of the survey instrument. The following are some of the most important considerations that can prevent problems and increase the likelihood of an effective instrument design (Cooper & Schindler, 2014; Fowler, 2009; Salkind, 2009; Zikmund et al., 2013).

• Keep questions simple, non-technical, and jargon free. Write to a 6th-grade reading level. Keep questions short and uncomplicated. Limit the number of questions to improve response rates. More questions normally translate into fewer responses.

• Avoid double-barreled questions. For example, do not ask participants to rate their level of job satisfaction and motivation in a single question. Satisfaction and motivation are two different concepts that cannot be measured by a single double-barreled question.

• Avoid leading questions and negative and positive terms that may sway the participant or suggest the correct answer.

• Avoid the use of subjective words, such as good, bad, fair, often, seldom, rarely, close, and far, since they mean different things to different people.

• Account for all possible answers and avoid “other, no opinion, or neither agree nor disagree” responses.

• Stay clear of emotionally charged or divisive language to avoid alienating participants.

• Seek exact information from questions. Asking “How old were you on your last birthday?” is much more exact than asking “How old are you?” With the latter, some participants will use their age as of the next birthday.

RCH 5301, Research Design and Methods 9

UNIT x STUDY GUIDE

Title

• Intersperse check questions and reversed score questions throughout the survey to verify the consistency of participant responses. These types of questions can be used to identify participants marking random answers or providing answers they think the researcher wants.

• Organize questions strategically. Opening questions should capture interest and be easier for participants to respond to. Similar questions should be grouped together. General questions should precede specific questions. Demographic questions should be placed at the end of the survey.

• Pretest the survey instrument yourself by taking it from the perspective of the participant. It is also useful to ask friends, family, or colleagues to review the survey for soundness, clarity, and flow. Finally, ask experts in survey design and/or the domain of interest for their input.

• Pilot test the survey instrument by administering it under survey conditions to a small, but representative, sample of the population to assess any problems in advance of full implementation of the survey. Pilot testing confirms survey validity and ensures the questions measure what they are intended to measure. In addition to validity, pilot testing should also elicit feedback from participants on soundness, clarity, flow of the questionnaire, motivation, and their overall experience in completing the instrument.

• Format the questionnaire to make it user friendly and easy to understand and complete. Bell et al. (2022, p. 234) provide guidance on how to optimally format a questionnaire.

• Make the questionnaire interesting to create a positive experience for participants.

(QuestionPro, n.d.)

In Closing

Like in all activities related to research, ethics must be a top priority in survey research. Survey research is not immune to the potential for abuse. One common form of unethical behavior related to survey research is when survey results are captured, but the sample size is not sufficiently large and/or the sampling frame does not adequately represent the population. In cases like these, it would be inaccurate and unethical for an overzealous researcher to generalize the results to the population of interest. This is especially pernicious since results are rarely peer-reviewed but may be used to make high-stakes decisions (Price & Mueller, 1986). Another serious ethical failing that is common in organizational research is the purposeful breach of confidentiality in making decisions about individual participants (Price & Mueller, 1986). Although the researcher may come under extreme pressure from management to divulge personal information related to survey results, it is their professional and ethical responsibility to maintain confidentiality and anonymity for the sake of participants. When in doubt, researchers should remember the adage, “First, do no harm.”

RCH 5301, Research Design and Methods 10

UNIT x STUDY GUIDE

Title

References Aiman-Smith, L., & Markham, S. K. (2004). What you should know about using surveys. Research

Technology Management, 47(3), 12–15. https://libraryresources.columbiasouthern.edu/login?url=https://search.ebscohost.com/login.aspx?dire ct=true&db=bsu&AN=12987072&site=ehost-live&scope=site

Annenberg Learner. (2013). Samples and surveys: Against all odds—Inside statistics [Video]. Films on

Demand. https://libraryresources.columbiasouthern.edu/login?auth=CAS&url=https://fod.infobase.com/PortalPl aylists.aspx?wID=273866&xtid=111536

Bell, E., Bryman, A., & Harley, B. (2022). Business research methods (6th ed.). Oxford University Press.

https://online.vitalsource.com/#/books/9780192640505 Borg, W. R., & Gall, M. D. (1979). Educational research (3rd ed.). Longman. Cooper, D. R., & Schindler, P. S. (2014). Business research methods (12th ed.). McGraw-Hill. Creswell, J. W., & Creswell, J. D. (2022). Research design: Qualitative, quantitative, and mixed methods

approaches (6th ed.). SAGE. https://online.vitalsource.com/#/books/9781071817964 Fields, D. L. (2002). Taking the measure of work: A guide to validates scales for organizational research and

diagnosis. SAGE. Fowler, F. J. (2009). Survey research methods (4th ed.). SAGE. Price, J. L., & Mueller, C. W. (1986). Handbook of organizational measurement. Pitman. QuestionPro. (n.d.). 10 steps to a good survey design [Graphic].

https://www.questionpro.com/features/survey-design/ Rosenthal, R., & Rosnow, R. L. (1975). The volunteer subject. Wiley. Ross, K. C., Clark, L. D., Padgett, T. C., & Renckly, T. R. (2002). Air University sampling and surveying

handbook. Department of the Air Force. Salkind, N. J. (2009). Exploring research (7th ed.). Pearson Education. Seashore, S. E., Lawler, E. E., & Cammann, C. (1983). Assessing organizational change: A guide to

methods, measures, and practices. Wiley. Zikmund, W. G., Babin, B. J., Carr, J. C., & Griffin, M. (2013). Business research methods (9th ed.). Cengage

Learning.