case study

Phase 2: Gather and Sort the Pieces As you experienced in the previous chapter, the creation of a survey question is not a haphazard or arbitrary process. It is exacting and requires paying significant

attention to detail and simultaneously addressing numerous issues. Whether you

create, borrow, or license a question, you generate specific measurement questions considering subject content, the wording of each question (influenced by the

degree of disguise and the need to provide operational definitions for constructs

and concepts), and response strategy (each producing a different level of data as

needed for your preliminary analysis plan).

In Phase 2, (see Exhibit 12-2) the researcher gathers the measurement questions and non-question elements in preparation to create a measurement instrument with

appropriate scope (topics) and coverage (questions within topics). While in the

previous chapter the focus was on measurement question design, here we focus on making sure we have a collection of the appropriate questions for fulfilling the

research objective. A measurement instrument needs three categories of

measurement questions and several non-question elements:

Administrative questions.

Classification questions.

Target questions.

Non-question elements.

Page 301

The non-question elements include the introduction, any transitions, instructions,

and a conclusion. We discuss these in the next major section.

The measurement instrument should accomplish the following:

Exhibit 12-2 Instrument Design Phase 2: Gather and Sort the Pieces

Encourage an individual to participate.

Help establish and maintain rapport with the participant.

Encourage a participant to provide accurate responses.

Encourage a participant to provide an adequate amount of information.

Discourage a participant from refusing to answer specific questions.

Discourage a participant from early discontinuation of participation.

Leave the participant with a positive attitude about survey participation.

Question Types

Page 302

Administrative Questions Administrative questions identify the (1) participant, (2) interviewer, (3) interview or research activity location, and (4) research conditions:

Not all administrative questions are asked but all are recorded; they are necessary to study patterns within the data and identify possible error sources. This

information becomes part of each case’s data record. Free-response (open-ended)

1. Researchers assign each participant a unique identification number; this

becomes their case number in a data file. With current technology, this might be

in the form of an electronically read code (e.g., barcode) that is scanned during the data entry process. It might also be the number that is used by the

participant to access a computer-based survey. Other administrative questions

about participants may include an assessment of the presence of children or other adults (personal interview), re-contact time (phone surveys), willingness

or reluctance, etc. Administrative questions often contain contact information

used later for follow-up clarification: an address, email address, or phone number.

2. Interviewers also are assigned unique identification numbers,

information about their gender, age, and ethnicity might be collected when the survey is conducted in person or 2. by phone.

3. Depending on study methodology, interview location might include city, type of

environment (e.g., mall, theme park, sports arena) and location within the environment (e.g., outside Macy’s at Seminole Mall for intercept studies),

whether the study is conducted inside or outside, and the methodology (phone,

personal interview, mobile or computer self-administered survey, etc.). When research requires a participant to do additional activities, the location of these is

also noted.

4. Research conditions include such items as day of week, time of day, weather,

congestion level, other events or situations occurring at the time of the survey,

distractions, etc.

questions may be the most efficient way to structure many of these questions, but

dichotomous choice, multiple-choice, or checklist questions may also be used.

Classification Questions Classification questions allow participants’ responses to target questions to be grouped so that patterns are revealed can be studied and tested, and insights can be

discovered. Classification questions cover demographic, economic, geographic, and

sociological variables. The topics covered by such questions include age, education, family composition, marital status, income, employment status and type of

employment, social class, ethnicity, language, political affiliation, etc. This group

may also include questions about behavior and attitudes (charitable giving, environmental sustainability, use of ride-sharing services, etc.) that are not the

focus but seem related to the target questions. The choice of classification

questions differs for each measurement instrument, as each one must serve a purpose in analyzing the data for a specific research objective. Sometimes such

questions may ask for sensitive information and may seem intrusive or unnecessary

to a participant, so researchers are careful to explain how they are used.

Some classification questions may appear early in the measurement instrument; these screen questions determine whether a participant has the requisite level of

knowledge or experience to participate. Some classifications questions, when used

within the instrument, serve as filter questions; they determine whether a participant is asked one or more questions within a target question topic or is

excluded from answering. In order to engage the participant early in the core issues

of any survey, many instrument designers place the bulk of classification questions in a separate section at the end of an instrument.

Classification questions may be free-response (e.g., age), dichotomous choice (e.g.,

gender), multiple-choice (e.g., income, education, employment, race, language, type

housing), or rating questions (e.g., degree to which you recycle).

Page 303

Target Questions Target questions address the investigative questions of a specific study. Of the three categories of questions, target questions represent the largest number and are the

most important. Researchers use these questions, in conjunction with classification

questions, to discover patterns that will help the manager make critical decisions.

To facilitate answering a survey, target questions are grouped by topics within an instrument. Based on the research objective, any hypotheses being tested, and the

investigative questions, the researcher determines the topics to be covered in the

instrument, the number of topics to be covered (instrument scope), and the number and breadth of questions needed in each group (instrument coverage). Researchers

may arbitrarily limit the scope and coverage of any particular instrument. When

the investigative questions form a long list, the strategy may be to survey participants using multiple instruments over time or select a separate sample so

that the survey time frame is not too long. Agile surveys do just this, keeping the

measurement instrument short.

>snapshot Energage identifies Top Workplaces

We all want to work for an employer that values us. And all employers want to attract and keep employees that are a perfect fit

and can help them achieve success. But what do you look for when

you are in the process of interviewing that might help you land a job with such an employer? One source you might choose is a published

survey on top places to work.

The Orlando Sentinel recently chose Energage as its research partner for one such survey. For the last 13 years, Energage has researched what distinguishes firms with high employee

engagement (called Top Workplaces) from those where employees

wished they were doing something else or doing it elsewhere. Top Workplaces have double or more the level of employee engagement

compared to the U.S. average (31%). “Top Workplaces…focus on

creating the right environment to unleash [employee] potential and

inspire performance,” states Doug Claffey, CEO, Energage. Retention

and recruitment are a problem in today’s labor market; increasing engagement is one solution. Employee engagement limits turnover,

boosts productivity, increases safety, enhances a firm’s reputation,

and supports a firm’s strategy. In its quest to understand the engaged workforce, Energage has collected more than 450,000,000

data points from more than 20,000,000 employees, working for

more than 58,000 organizations. During analysis, Engage can compare each new study to this vast repository of data.

Central Florida is one of the fastest growing areas of the United

States with a population in excess of 2.5 million. Orlando, Central

Florida’s largest city and home to Disney World and Universal Studios, is the most visited destination in the United States. The

tourism industry employs 41 percent of the workforce. Besides

tourism, advanced manufacturing, aviation and defense, life sciences and healthcare, and information technologies are its dominant

industries. For the Orlando project, The Orlando Sentinel asked employers or employees to nominate their organizations as Top Workplaces. Energage also reached out to businesses in the area. To

participate, employers needed at least 35 employees. If a firm had

been designated a Top Workplace in the prior year’s study, even with

fewer than 35 employees, it was included. Of the 826 employers identified and invited, 112 participated in the survey. These firms

collectively employ 74,337 people. Of these, 25,431 individual

employees participated. Participants could choose to take either a paper or online survey.

The survey’s questionnaire contained 24 questions covering seven

major topics. It took 10 minutes to complete:

Jupiterimages/Getty Images

Alignment: company’s direction, values, cooperation.

Connection: employee appreciation, meaningful work.

Effectiveness: working efficiently and well, sharing viewpoints, encouraging new ideas.

My manager: cares about concerns, helps employees develop.

Engagement: motivation, retention, recruiting.

Leadership: confidence in company leaders.

Basics: pay, benefits, flexibility, training, expectations.

Page 304

Employees consistently rated issues of connection and alignment

most important to them, while statements related to pay and benefits rated less important. Energage has discovered that one of the most

powerful differences in Top Workplaces is the firm’s use of

technology to empower the culture of the firm. “When done right, technology brings people together,” shared Claffey. “And that’s

important, because alignment is a key pillar for any high–performing

organization. But when done wrong, technology polarizes people.”

So when you are interviewing, look for firms that use technology to celebrate their culture and build real connections based on trust and

appreciation. And ask other employees about how the environment

supports engagement. But remember the sample is self-selected: there are many reasons a firm may have chosen not to participate.

www.energage.com; www.orlandosentinel.com

Target questions should minimize shifts in frame of reference without

notification. This is especially true of behavior questions that provide a time frame.

Participants often interpret one question in light of earlier questions and may miss shifts of perspective or subject unless they are clearly stated. Participants fail to

listen or read carefully and frequently jump to conclusions about the importance of

a given question before it is completely stated. Their answers are strongly influenced by their own experiences (their frame of reference). Any change in

subject or topic needs to be obvious. Most questionnaires that cover a range of

topics are divided into sections using topic labels with clearly defined transitions between sections to alert the participant to the change in frame of reference.

Target questions may be structured (present the participants with a fixed set of

choices; closed questions) or unstructured (provide the participant with a frame of

reference for their self-crafted answer; open-ended questions). Target questions are designed using the full range of measurement scales. To facilitate answering a

survey more quickly, instrument designers tend to use fewer question types within a

single measurement instrument; this allows the participant to get familiar with question structure.

Pretesting: Scope and Coverage It is during this phase that a researcher might collaborate with research colleagues. From these colleagues, a researcher may seek evaluation of an instrument’s scope

and coverage. Colleagues also can spot when questions assigned to one topic seem

better suited to a different topic, alerting a researcher to potential order problems. Or they may suggest a different order of topics than the plan proposes. Rather than

developing an instrument during this phase, the researcher provides a mock-up of

the question topics, including each question’s focus within a topic, and requests feedback. Obtaining feedback before an instrument is created saves valuable time,

as well as prevents bias and error.

A second round of pretesting might order the topics and provide the actual

questions for the topic, but not designated in any order. Seeing a question in the context of its topic for the first time might lead to recommendations about question

inclusion (drop or add). Of course, the colleague also might suggest that a question

should be asked in a different way or that a different scale would be more appropriate given the other questions within the topic.

Responsibility Fulfillment In this phase, the research has many responsibilities. Exhibit 12-3 summarizes how actions during this phase can fulfill those responsibilities.

Exhibit 12-3 Actions That Fulfill Phase 2 Responsibilities

Responsibilities Actions

Encourage an individual to

participate.

Help establish and maintain

rapport with the participant.

Encourage each participant

to provide accurate responses.

Encourage each participant

to provide an adequate amount of information.

Discourage each participant

from refusing to answer specific questions.

Provide appropriate introduction.

Provide information on anonymity and

confidentiality. Order high-interest topics early; order

sensitive topics at the end. Provide clear instructions.

Provide operational definitions as

necessary. Use common vocabulary in instructions and

transitions.

Order questions from general to specific. Use color/design features to enable

participant to see full range of tasks and

response options. Include prompts to complete a skipped or

missed question.

Establish rapport early. Provide a prompt when a question is

skipped or missed that stresses its

importance.

Responsibilities Actions

Discourage each participant

from early discontinuation of participation.

Leave the participant with a

positive attitude about survey participation.

Order topics based on participant interest. Keep all questions relating to the same topic

together. Show appreciation for the participant’s time

by restricting topics and questions to what is

critical. Provide a status bar on computer/mobile

surveys. Vary response strategy to avoid boredom. When using the same scale use repetitive

response categories, when possible. Use visual stimuli as part of question,

instruction, or transition. Use visual response options

(mobile/computer).

Include topics and questions that make the

participant feel knowledgeable, having

contributed to something important. Ask meaningful questions related to the

topics. Thank the participant in the conclusion.