615 assignment

Article

Why Do Web Surveys Take Longer on Smartphones?

Mick P. Couper 1

and Gregg J. Peterson 1

Abstract Surveys completed on mobile web devices (smartphones) have been found to take longer than surveys completed on a PC. This has been found both in surveys where respondents can choose which device they use and in surveys where respondents are randomly assigned to devices. A number of potential explanations have been offered for these findings, including (1) slower transmission over cellular or Wi-Fi networks, (2) the difficulty of reading questions and selecting responses on a small device, and (3) the increased mobility of mobile web users who have more distractions while answering web surveys. In a secondary analysis of student surveys, we find that only about one-fifth of the time difference can be accounted for by transmission time (between-page time) with the balance being within-page time differences. Using multilevel models, we explore possible page-level (question-level) and respondent-level factors that may contribute to the time difference. We find that much of the time difference can be accounted for by the additional scrolling required on mobile devices, especially for grid questions.

Keywords online surveys, web surveys, smartphone surveys, survey completion times

Introduction

A consistent finding in the emerging survey literature on mobile web surveys is that surveys

completed on mobile devices take longer time than those completed on a PC (desktop or laptop),

with tablets (when measured separately) occupying a middle position. This is important for survey

designers because mobile phones are also associated with higher breakoff rates (e.g., Guidry, 2012;

Jue & Luck, 2014; Mavletova & Couper, 2013, 2014; Sarraf, Brooks, & Cole, 2014). While several

authors have speculated on the causes of these differences, few researchers have attempted to

empirically explore the reasons behind these time differences. We use data from a campus-wide

survey conducted annually from 2012 to 2014, in which both server-side times and client-side times

were captured, along with other paradata, to explore some of the possible reasons for this time

differential.

1 University of Michigan, Ann Arbor, MI, USA

Corresponding Author:

Mick P. Couper, University of Michigan, Ann Arbor, MI, USA.

Email: [email protected]

Social Science Computer Review 2017, Vol. 35(3) 357-377 ª The Author(s) 2016 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/0894439316629932 journals.sagepub.com/home/ssc

In the following sections, we first review the literature on the topic and explore the possible

mechanisms for the time differences observed. We then describe the data and the analytic strategies

we employed, before summarizing the results of our analyses and offering suggestions for future

research in this area.

Background

Our primary focus in this article is on web surveys completed using a browser, whether on a PC

(desktop or laptop computer), a tablet, or a mobile device (smartphone). We are not concerned with

app-based surveys or those designed solely for mobile devices. A key characteristic of browser-

based surveys is the use of the Internet for transmission of survey pages from the server to the client

(the respondent’s device) and the transmission of the respondent’s answers in turn from the client to

the server. The survey instrument may or may not be optimized for mobile devices (see Buskirk &

Andrus, 2012). Respondents may choose to use mobile devices to complete the surveys whether

intended by the survey designer or not or, in recent experiments, be randomly assigned to take the

survey on a particular device. In most of these cases, respondents are using their own devices, rather

than being provided with devices for survey completion.

A consistent finding across surveys is that respondents who complete the survey on a mobile

device take longer on average than those who use a PC. Several early studies observed longer

completion times for the so-called unintentional mobile users (de Bruijne & Wijnant, 2014; Peter-

son, 2012), those who used a smartphone to complete the survey without prompting from the survey

administrator and without the survey being optimized for smartphone use (see Chrzan & Saunders,

2012; Cook, 2014; Horwitz, 2014; Hupp, Schroeder, & Piskorowski, 2014; Jue & Luck, 2014;

Lambert & Miller, 2014; Lugtig & Toepoel, 2015; Mavletova, 2013; Maxl & Baumgartner, 2013;

McClain, Crawford, & Dugan, 2012; McGeeney & Marlar, 2013; Pape & Barron, 2013; Peterson,

Mechling, LaFrance, Swinehart, & Ham, 2013). In addition, Peterson (2012) examined completion

times for 17 different nonoptimized web surveys and found survey length to be consistently longer

(by ratios of 1.25 to 1.50) on mobile devices. The proportion of mobile users varied across all these

studies, as did the topic and length of the survey and the populations surveyed.

One explanation offered for the time difference is that these surveys were not optimized for

mobile devices. However, a number of studies have found longer completion times for mobile-

optimized surveys than for surveys completed on PCs, again with respondent self-selecting the

device (see McGeeney & Marlar, 2013; Sommer, Diedenhofer, & Musch, 2015; Wells, Bailey, &

Link, 2014). Finally, several studies randomly assigned respondents to use either a PC or a mobile

device (smartphone) to complete the survey. In most of these cases, the mobile version was opti-

mized for smartphones (see Antoun, 2015; de Bruijne & Wijnant, 2013; Fischer & Bernet, 2014;

Lugtig & Toepoel, 2015; Mavletova & Couper, 2015), while in one case it was not optimized for

mobile use (Mavletova & Couper, 2013). Two studies compared optimized versus nonoptimized

versions of the instrument (e.g., McGeeney & Marlar, 2013; Peterson et al., 2013), and both found

the former to be faster than the latter, but still slower than completion on a PC.

All of the above studies found longer response times for mobile device or smartphone users than

for PC users, with time ratios ranging from a low of 1.11 to a high of greater than 3 for nonoptimized

surveys. Even in those studies where mobile optimization was used and respondents were randomly

assigned to device, mobile completion times were slower than PC completion times by ratios of

1.02–1.78.

Three studies found equal or lower completion times for smartphones than for PCs. Toepoel and

Lugtig (2014) report mean completion times of 250 s for mobile users and 245 s for PC users, a

nonsignificant difference, t(433) ¼ �0.31. Buskirk and Andrus (2014) prescreened opt-in panel members for iPhone use. Panelists were then randomly assigned to receive a PC survey or an app-

358 Social Science Computer Review 35(3)

like mobile browser survey to complete on their iPhones. The completion rate was 2.6 times higher

for PC than for iPhone invitees, but completion times were significantly lower for iPhone users

(median of 8.25 min) than for PC users (median of 12.4 min). Similarly, Wells, Bailey, and Link

(2014) randomly assigned smartphone users to an app-based survey or a PC-based web survey. They

reported median completion times of 5.5 min for the mobile app version and 5.8 min for the PC web

version. Thus, with a few exceptions, the studies reviewed above find that surveys take longer time

to complete on mobile devices (particularly smartphones) than on PCs. This seems to be true for both

mobile-optimized surveys and those not optimized for smartphones. 1

Table 1 presents a summary of

these studies. The rows are ordered by the ratios of mobile to PC completion times. Formal meta-

analysis of these completion times would be difficult, as some authors report medians, while others

report means. Some include tablets with mobile devices, and others combine them with PCs.

Further, many other details of the sample and survey designs are missing.

In one additional study (not reported in Table 1), Gummer and Roßmann (2014) conducted a

multilevel model of survey completion times among 23,800 respondents nested within 21 surveys.

They do not report the relative mean completion times by device but found a significant positive

coefficient for survey length (i.e., longer times) for smartphone users (vs. PC), but not for tablet

users. This finding is consistent with those reviewed above.

All of these studies (with the exception of Gummer & Roßmann, 2014; Mavletova & Couper,

2015) look at overall completion times, not at item-level times. Further, while several speculate on

the likely reasons for the time differences, none attempt to measure these directly (again with the

exception of Mavletova & Couper, 2015). For example, Peterson (2012) speculated that ‘‘Extra

length appears to be more related to network latency than to survey complexity,’’ and called for more

analysis. Mavletova (2013) offered three possible reasons for the disparities in completion times she

observed. First, cell phones may have slower Internet connections and thus may take longer to

download questions (especially those with images). Second, participation via cell phone demands

more time and effort to answer questions on a small screen without using a mouse and keyboard

(Zahariev, Ferneyhough, & Ryan, 2009). In her study, significantly more mobile respondents

reported difficulty completing the survey (9.5% in mobile vs. 0.6% in PC web). Third, cell phones allow the respondents to be more ‘‘mobile’’: Whereas only 3% of PC respondents completed the survey outside the home or office, almost 20% of mobile respondents did so. de Bruijne and Wijnant (2013) offered similar explanations and did Jue and Luck (2014) and Gummer and Roßmann (2014).

In an examination of item-level times, Mavletova and Couper (2015) found that the average

network latency, which is the time to deliver the next page after clicking the ‘‘Next’’ button, was

twice as long on mobile phones than on a PC: 3.9 s and 1.9 s, respectively, t(2078) ¼�13.44, p < .001. A linear regression, predicting the completion time on mobile phones showed that the screen

height, level of education, and the reported type of Internet connection were significant predictors.

Those with higher education completed the survey almost 1 min faster than those without higher

education (p < .05). Each additional 100 pixels in screen size decreased the predicted completion

time by 0.2 min (p < .05), while those who used 2G or 3G Internet connections took more time to

complete the survey, by 3.4 min (p < .001), than those who used Wi-Fi.

This article is an attempt to partially address this gap in the literature and explore the reasons for

these differential times using a secondary analysis of an existing data set with detailed item-level

response times.

Possible Reasons for Response Time Differences

The papers reviewed above offer a number of possible explanations for the response time differences

between smartphone and PC completion. We review these again below, along with additional

reasons for differences observed.

Couper and Peterson 359

Connection Speed/Quality

Several of the papers point to network latencies—the amount of time it takes for the server to process

a request and transmit the web page to the client and to transmit the respondent’s answer back to the

server—as reasons for the slower mobile times. Some cite a paper by Comer and Saunders (2012),

who report that mobile devices have 35–65% longer latencies (as measured by the Strangeloop latency comparison test) but don’t provide further details on the results of the test.

Mavletova and Couper (2015) report that transmission time accounted for about 28% of the time difference in their survey. They also report that those using cell connections (2G or 3G) took longer to

complete the survey than those using Wi-Fi. But tablet computers have similar connections to smart-

phones and should be just as slow. The fact that tablet completion times are closer to those of PCs

suggests that transmission speed or quality cannot account for all of the time difference. Ideally, we

would measure connection speeds or type of connection passively during the survey. Alternatively, we

could ask respondents what type of connection they were using. In our analysis, we measure this

indirectly by comparing server-side (or between-page) times with client-side (or within-page) times.

Screen Size

The size of mobile devices is a frequently mentioned source of the differences in completion times.

This argument has two parts. The first is that questions are not fully visible on mobile screens and

Table 1. Completion Times (in Seconds) for PC and Mobile Users (Ordered by Time Ratios).

Source PC

Time Mobile Time Ratio

Self-Selected (SS) or Random Assignment (RA)

Mobile- Optimized

Mavletova (2013), long survey 9.66 29.46 3.05 SS No Mavletova (2013), short survey 5.28 15.46 2.93 SS No Mavletova and Couper (2013) 9.1 20.5 2.25 RA No Antoun (2015) 9.8 17.4 1.78 RA Yes Wells, Bailey, and Link (2014) 5.3 8.9 1.68 SS Yes Fischer and Bernet (2014) 8.5 13.8 1.62 RA Yes De Bruijne and Wijnant (2013) 5.65 9.04 1.60 RA Yes Pape and Barron (2013) 8.2 13.9 1.56 SS No Peterson et al. (2013) 7.1 11.0 1.55 RA No Sommer, Diedenhofen, and Musch (2015) 24.1 37.3 1.55 SS Yes Chrzan and Saunders (2012) 2.6 3.77 1.45 SS No Jue and Luck (2014) 7.53 10.46 1.39 SS No Mavletova and Couper (2015) 15.8 21.8 1.38 RA Yes Hupp, Schroeder, and Piskorowski (2014) 32.5 44.0 1.35 SS No Cook (2014) 29 39 1.34 SS No McGeeney and Marlar (2013) 2.43 3.11 1.28 SS No Peterson et al. (2013) 7.0 8.9 1.27 RA Yes Horwitz (2014) 37.4 45.4 1.21 SS No McClain, Crawford, and Dugan (2012) 24.23 28.73 1.19 SS No Lambert and Miller (2014) 27.6 31.8 1.15 SS No McGeeney and Marlar (2013) 2.43 2.795 1.15 SS Yes Maxl and Baumgartner (2013) 6.32 7.0 1.11 SS No Lugtig and Toepoel (2015) 13.88 14.33 1.07 SS No Toepoel and Lugtig (2014) 4.08 4.17 1.02 RA Yes Wells et al. (2014) 5.8 5.5 0.95 RA Yes (app) Buskirk and Andrus (2014) 12.4 8.25 0.67 RA Yes (app)

360 Social Science Computer Review 35(3)

that users have to scroll (or pinch and zoom) to read the questions. If this is the case, we would

expect smaller time differences for short yes/no questions where scrolling is not needed but bigger

differences for longer questions (especially grid or matrix questions). While we did not measure

smartphone user actions such as pinching and zooming, we do have an indicator for scrolling (and

the amount of time taken to scroll) in our survey.

A second explanation is that reading on mobile devices is slower because of the smaller font

sizes. If this is the case, we should expect shorter questions to take longer times on mobile

devices too.

Input Method

A third explanation offered is that it takes longer to select a response on a mobile device. If this is the

case, then we would expect longer response times for nonoptimized surveys (where the radio buttons

and check boxes present a smaller target for selection) than for mobile-optimized surveys (where a

key feature is bigger buttons for easier selection). We should also expect to see bigger time differ-

ences for open-ended questions, where it may be slower to type) than for closed-ended questions.

While several studies have examined the length of open-ended responses by device used (e.g.,

Mavletova, 2013; Peterson, 2012; Toepoel & Lugtig, 2014), none (to our knowledge) have looked

at response times to such questions (conditional on the length of response).

Multitasking

Some authors have argued that mobile device users are more likely to multitask and therefore may

not respond as quickly. Lynn and Kaminska (2013) make a distinction between aural or visual

distractions (which they argued were more prevalent in mobile phone surveys) and multitasking

(carrying out other tasks simultaneously or sequentially). Zwarun and Hall (2014) similarly make a

distinction between environmental distractions (e.g., background noise), nonmedia multitasking

(e.g., having a conversation with someone or other task-switching activities), and electronic media

multitasking (e.g., checking e-mail and updating Facebook status), which could be on the same

device or on another device. Multitasking on the same device could be measured passively through

paradata (e.g., JavaScript OnBlur or OnFocus functions, which indicate that the respondent has left

the survey page; see Sendelbah, Vehovar, & Slavec, 2014). Nonmedia multitasking or multitasking

on a different device could be measured by looking at response times (with times above a certain

threshold being used as an indicator of possible multitasking), but this presents a potential endo-

geneity problem for our research. Another approach is to use self-reports, whether directly (e.g.,

what were you doing while completing the survey) or indirectly (e.g., where were you while

completing the survey).

Using the latter approach, Mavletova and Couper (2013) found that mobile users were more

likely to report completing the survey away from home or office (43.8% vs. 23.0% for PC users in Wave 1) and more likely to report the presence of others (30.9% vs. 17.3% in Wave 1). Antoun (2015) used both approaches and found that smartphone users were more likely to report multi-

tasking while completing the survey (54.3% vs. 44.4% for PC users) and more likely to report completing the survey away from home or work (7% vs. 1.6%). Similarly, Lorch and Mitchell (2014) reported that 26% of mobile users and 18% of PC users admitted doing other things while completing the survey. However, these differences in self-reported distraction or multitasking may

not be sufficiently large to account for the time differences. We do not have a measure of distraction

or multitasking in our survey but suggest these are worth including in other studies comparing

smartphone and PC survey behavior. We are unable to detect constant distractions (e.g., background

Couper and Peterson 361

noise and watching TV), but if the distractions are intermittent, we should expect greater variability

in response time across items for respondents using mobile devices.

Familiarity or Comfort With Device

A final reason offered for differences in response times relates to the relative comfort or familiarity

with the device. If this were the case, those choosing to use their own devices may be assumed to do

so because of greater familiarity or ease with the device. In contrast, those randomized, encouraged,

or incentivized to use a smartphone (i.e., ‘‘intended’’ smartphone users) may be less familiar with the

devices. However, we still find response time differences in those studies (including the data we

analyze here) where respondents chose the device they used to complete the survey. But this reason

may be explored further in online panels (such as the LISS panel; see http://www.lissdata.nl/), where

a subset of panelists was provided with devices, while others used their own devices. Comparing

response times by self-reported familiarity or frequency of smartphone use (in surveys where this is

ascertained) would also be useful.

We are unable to explore all of these possible mechanisms here. However, in this article, we

examine item-level response times in a secondary analysis of web survey data where some chose to

use smartphones to attempt to disentangle some of these possible causes of time differences between

mobile (specifically smartphone) and PC users. 2

We describe the data sources in the next section,

before proceeding to describe the analysis methods and results.

Data and Method

We analyze data from 3 years of the Sustainability Cultural Indicators Program (SCIP), a multiyear

survey of University of Michigan students, staff, and faculty focused on behaviors and attitudes

related to environmental sustainability (see http://graham.umich.edu/leadership/scip; Marans &

Callewaert, 2015). The survey has been conducted each year since 2012. DatStat’s Illume data

collection system (see http://www.datstat.com/) was used for all survey years. We focus only on

the student surveys here.

Survey Design

The 2012 and 2013 surveys were developed using Illume’s default mobile style sheet. This style

sheet uses bigger fonts and input controls (radio buttons and check boxes) and bigger navigation

buttons for mobile devices. The input types were consistent across devices. Grids are not changed.

This could be characterized as ‘‘mobile friendly’’ or ‘‘partially optimized.’’ In the 2014 survey, a

custom style sheet was used that changed the grid items into single items presented vertically on the

same page on mobile devices, in addition to the other features described above. In all years, a single

question or grid was presented on each page.

In the 2012 survey, there was no instruction regarding mobile device use. Given the higher

breakoff rate for smartphone users found in the 2012 survey (see Table 2), the welcome page for

the 2013 survey included the following warning: ‘‘This survey is best viewed and completed on a

desktop, laptop, or tablet.’’ This statement was removed for the 2014 survey.

The survey instruments were not identical across the 3 years but, given that a key focus of SCIP

was to examine trends in attitudes and behavior, a large proportion of the items are similar across the

three surveys. This permits us to examine individual sets of items that were unchanged across the

three rounds of the survey.

362 Social Science Computer Review 35(3)

Data Collection

The 2012 survey was conducted in October and November of that year. A sample of 11,000 full-time

students was drawn from the registrar’s database. An embedded experiment compared paper versus

e-mail prenotification, which had no significant effect on outcomes. Following prenotification, an

e-mail invitation with a link to complete the survey was sent. Two e-mail reminders were sent to

nonrespondents. A total of 5,021 students clicked on the survey uniform resource locator (URL), of

which 4,723 made it past the informed consent screen and started the survey. Of these, 4,018

completed the survey (for a calculated response rate of 36.5%). Breakoff rates were significantly higher for smartphones, 27.7% versus 13.4%, w2(1) ¼ 73.3, p < .0001 (see Table 2).

The sample for the 2013 survey included 2,867 students who had participated in the 2012 survey,

plus a fresh sample of 13,000 undergraduate and 1,500 graduate students, for a total sample of

17,367. All sample persons received an e-mail prenotification, followed by an e-mail invitation, and

up to two reminders (also by e-mail). We have data on a total of 3,730 respondents, including 3,223

completes (for a response rate of 18.6%). The breakoff rate was again significantly higher for smartphones than for PCs, 29.6% versus 11.5%, w2(1) ¼ 108.7, p < .0001.

The 2014 survey invitation was sent to a sample of 16,091 students (including 2,750 who were

interviewed in 2012), again stratified by class. All sample persons received an e-mail prenotifica-

tion, a subsequent e-mail invitation (with the URL and login to the survey), and up to three e-mail

reminders. We have data on a total of 4,935 respondents, including 4,233 completes, for a response

rate of 26.3%. Again, smartphone users had a significantly higher breakoff rate than PC users, 18.4% versus 13.0%, w2(1) ¼ 21.3, p < .0001 (see Table 2).

Timing Data

We have two sources of time data. The page-level server-side times, measured by the Illume

software, capture the elapsed time from the transmission of the page from the server to the receipt

of the data from that page back at the server and the transmission of the following page. That is,

PageStartTime for page nþ1 is same as PageEndTime for page n. A page consists of one question or multiple grid items. Thus, the page times (measured in seconds) include response times (within-page

times), transmission times, and server processing times (between-page times). Our second source of

time data comes from JavaScript paradata code embedded in each of the survey pages (see, e.g.,

Heerwegh, 2003, 2011). This captures the time (in milliseconds) from when the page is loaded on the

client (respondent’s browser) to when the data are sent from the browser to the server, along with

elapsed time to all interim events (e.g., mouse clicks, keyboard use, and scrolling actions). See

Kaczmirek (2009, p. 83) for a fuller description of the various time components. The difference

Table 2. Key Outcomes for the Three Sustainability Cultural Indicators Program Surveys.

Key Outcomes 2012 2013 2014

Sample size 11,000 17,367 16,091 Number of starts 5,021 3,730 4,935 Proportion starting on a smartphone 10.8% 12.7% 23.8% Number of completes 4,018 3,223 4,233 Response rate 36.5% 18.6% 26.3% Breakoff rates

Overall 14.9% 13.6% 14.2% PC 13.4% 11.5% 13.0% Smartphone 27.7% 28.6% 18.4%

Couper and Peterson 363

between the total time and the within-page time is thus the between-page time. Given that the server

times are measured in seconds and the client times are measured in milliseconds, there are some

rounding errors in subtracting one from the other. But these give us a reasonable approximation of

the two key time components. From the client-side paradata, we can further identify the elapsed time

to the first selection (sometimes referred to as response latency). For ease of interpretation, we

convert all times measured to seconds.

Respondents may visit a page multiple times, for example, if they backed up to review or change

a previous answer. This is relatively rare, so we include all page visits. A sensitivity analysis looking

only at the first visit to each page did not change the results substantially. We similarly include both

complete and partial interviews (breakoffs) in our analyses. Restricting the analyses to completed

cases only again did not lead to changes in our conclusions.

In addition to the time variables, we coded characteristics of the survey pages. Using MS Word’s

word count feature, we captured the length of each survey page (in words) as well as the number of

words in the question stem (up to the first response option). We coded the number of individual

questions on each page and the question types (e.g., grid, single selection, and open-ended). For

grids, we have the number of rows (items) and the number of columns (response options). With the

exception of grid questions, all pages consisted of a single question. Each of these characteristics

was consistent across both PC and mobile versions of the survey.

Key respondent-level factors in response times are education, age, and Internet experience (see

Couper & Kreuter, 2013; Yan & Tourangeau, 2008). Given the relative homogeneity of the student

population we study on these characteristics, our primary focus is on page-level characteristics and

device as factors in response time.

Results

Our primary focus was on the results of the 2012 survey. We subsequently examine the 2013 and

2014 surveys and report noteworthy results from those surveys below later.

2012 Student Survey

First, we need to determine what device respondents used to complete the survey. This is extracted

from the user agent string (see, e.g., Callegaro, 2010). We find that 509 of the 4,723 respondents (or

10.8%) used a smartphone to start the survey, a further 59 used a tablet (1.2%), while 4,155 used a desktop or laptop PC (88.0%). Given the small number of tablet users and the evidence reviewed above that tablets behave very similarly to PCs in terms of performance on web surveys, we combine

them with the PC group. A small number of respondents switched devices: 1.4% from a smartphone to a PC and 0.06% from a PC to a smartphone. Given this, we use the device of the initial login as a proxy for that used throughout the survey.

As with any paradata, the time data we analyze are messy (see, e.g., Yan & Tourangeau, 2008).

We have a number of outliers (e.g., pages that took more than 24 hr to submit), along with some

negative times and a small number of pages where paradata was not captured. Our raw data file

contained information on over 300,000 page visits for 4,723 respondents (4,214 completes and 509

partials). After extensive cleaning, in which we truncated extreme values at the 99th percentile,

removed corrupted or missing paradata files, and so on, we are left with 299,009 observations (page

visits) with valid page times, nested within 4,585 respondents, with an average of 65.2 page visits per

respondent. The subsequent analyses focus on the page visit level.

The next question is whether there is indeed a time difference between PC and smartphone users

at the page level. Preliminary analyses (without accounting for the clustering of pages within

respondents) show a mean time of about 15.0 s per page for PC users and 17.3 s for smartphone

364 Social Science Computer Review 35(3)

users (a ratio of 1.18). The within-page mean times are 14.6 and 16.4 s, respectively, and the

between-page mean times are 0.39 and 0.96 s, respectively. This suggests that about 25% of the difference in time is accounted for by between-page time differences (transmission times or network

latency), while the balance is within-page time (time spent answering the survey questions, includ-

ing any multitasking). This is similar to Mavletova and Couper’s (2015) finding of about 28% of time being between-page time.

Given that most of the time—and most of the difference in time—is time spent answering survey

questions, this suggests that slower transmission times associated with smartphones account for a

relatively small fraction of the longer completion times observed in the literature. Our next step is to

use multilevel models to explore the differences in within-page times by device, using characteris-

tics of the survey items (pages) to understand what accounts for such differences.

The time measures are positively skewed, so we also fit log-transformed versions of these

models. These yield similar results to the untransformed versions, so we present the results of the

original variable here for ease of interpretation. The coefficients can thus be directly interpreted as

effects on time (in seconds). We use unweighted multilevel linear mixed models (using SAS PROC

MIXED) to explore the effect of device and page characteristics on response times.

We first fit an unconditional (null) model to ascertain the proportion of variance accounted for

by pages and respondents, respectively, in a crossed random effects model. This model can be

specified as:

TIMEpr ¼ b0 þmr þ mp þ epr;

where TIME is the page-level time measure (within-page time in this case), mr is the random effect associated with the respondent, mp is the random effect associated with the page, and epr is the residual variability associated with each page p and each respondent r. All random effects are

assumed to follow a normal distribution.

Using this model, we estimate the intraclass correlation coefficient (ICC) for each level of the

model, as follows:

ICCr ¼ s2r

s2r þ s2p þ s2 for respondents and ICCp ¼

s2p s2r þs2p þs2

for pages:

Based on the ICCs, we estimate that about 8.7% of the total variation in page times is due to between-respondent variation, while about 31.0% of the total variation is due to variation between pages. That is, variability between survey pages accounts for more of the variation in time than

variability between respondents. The intercept for this null model is 14.8 s per page.

A second null model (again using crossed random effects) focused on the between-page (or

transmission) times. Here the ICC for respondents is 0.381, while that at the page level is 0.0018.

Thus, over a third of the variation is at the respondent level. This suggests that the between-page

(transmission) times vary less between survey pages, as we would expect: Aside from small differ-

ences in page sizes, transmission times should be relatively constant within respondents.

For technical reasons (exceeding memory limits and estimating degrees of freedom), the crossed

random effects models could not be estimated when adding predictors to the models. The remaining

analyses are based on nested models (with pages are nested within respondents). While the estimated

coefficients from the two types of models differ slightly, the overall conclusions from the two sets of

models are very similar.

Returning to our primary focus on within-page time variation, our next goal is to try to account

for this variation by controlling for fixed characteristics of the pages. The first step in this process is

to check whether the effect of device is still significant in the multilevel model, by adding this

variable as the single page-level covariate. The coefficient for device (1 ¼ smartphone, 0 ¼ PC) is

Couper and Peterson 365

statistically significant (p < .0001), but device explains very little (<1%) of the variation in within- page response times. The estimated slope coefficient is 2.094, which suggests that smartphone users

take an average of just over 2 s longer to answer a question (14.67 s for PC users and 16.76 s for

smartphone users). This conforms to the earlier crude page-level analyses. We then added a variable

to examine whether this differs by the type of browser used (e.g., Firefox and Safari). Browser has no

significant main effect on within-page response times and does not interact with device. However,

we retain browser as a control variable in all subsequent models.

To what extent does within-page completion time vary by the type of question? Table 3 shows the

mean within-page time in seconds by several key types of questions (information questions are those

where a respondent simply presses ‘‘next’’ to continue without entering a response). The table

includes both the raw (unadjusted) means, and the least squares adjusted means from a multilevel

model controlling for browser type. In this model, the interaction of question type and device is

statistically significant (p < .0001), as are the main effects for device and question type. We see that

open questions take longer to answer than questions requiring the selection of a response option.

Grid questions take longer to complete than all other question types. We also see that across all

question types, smartphone users take longer than PC users.

However, some of the question types are relatively rare in the SCIP survey. For instance, single-

choice questions make up 46.2% of all page visits, grids make up 34.5%, only 7.2% of pages include an ‘‘other specify’’ option, 6.8% involve numeric entry (e.g., number of people, zip code, and age), 3.8% involve text entry, while only 1.4% are information items. Given that prior research has identified grids as particularly problematic for mobile device users (e.g., de Bruijne & Wijnant,

2013; McClain & Crawford, 2013; Peterson et al., 2013), we collapse these question types into an

indicator for pages with grids versus all other question types.

As noted earlier, we have an indicator from the paradata on whether the respondents scrolled

(whether horizontally or vertically) on each page while responding to the questions. We also have

the time taken to scroll, measured from the beginning of each scrolling event to the beginning of

the next event (e.g., selecting a response). The percentages of pages with scrolling observed by

device are presented in Table 4, along with the scrolling behavior for grid questions. Overall, we

see much more scrolling by smartphone users (49%) than PC users (4%). Much of this is vertical scrolling, but horizontal scrolling also occurs more among smartphone users. Grid questions

involve significantly more scrolling overall, but particularly so for smartphone users. Even on

the nongrid pages—which contain only a single question—smartphone users scroll about a third of

the time. Given a minimum readable font size on a smartphone, we expect that the smaller the

display, the more scrolling will be needed.

Given that scrolling is more prevalent in grid question than in other question types and that

scrolling is more frequent among smartphone users, to what extent does the need for scrolling

Table 3. Mean Within-Page Times (in Seconds) by Question Type and Device, 2012 Survey.

Question Type

Unadjusted Means LS Means from Nested Multilevel Model

PC Smartphone PC Smartphone

Single choice 8.22 10.38 8.28 10.64 Single choice with ‘‘other specify’’ 9.35 10.14 9.43 10.54 Numeric entry 6.45 7.80 6.58 8.26 Text entry (open question) 19.56 23.01 19.70 23.51 Grid 25.82 27.44 25.93 27.72 Information 5.05 5.15 5.12 5.61

Note. LS ¼ least squares.

366 Social Science Computer Review 35(3)

explain the differences in response times? Table 5 shows the mean times from multilevel models

with device as the only covariate. All differences are statistically significant (p < .0001). From the

first row of the table, we see that smartphone respondents spend more time scrolling than PC

respondents. Given that only about 9% of pages overall have any scrolling (see Table 4), the second row restricts the analysis to those cases with scrolling (i.e., eliminating all cases with scrolling time

¼ 0). Again, we see that smartphone respondents spend more time scrolling. When a respondent does scroll, they take an estimated average of 8.4 s per page to scroll on smartphones and 5.6 s on

PCs, about 50% longer. The final row in Table 5 subtracts the time spent scrolling from the total within-page time. Here we see that when scrolling time is removed, smartphone respondents appear

to take less time to answer than PC respondents. In other words, the longer within-page times

experienced by smartphone users appear to be largely due to the need to scroll and the time taken

to do so. This is particularly true of grid questions.

We explore this further with two nested models. The first model includes main effects for device

type (1 ¼ smartphone, 0 ¼ PC), browser, a grid flag (1 ¼ yes, 0 ¼ no), a count of the number of words on the page, an indicator for horizontal or vertical scrolling (1 ¼ yes, 0 ¼ no), and the scrolling time (for those pages with scrolling). The second model adds the interaction of device

and grid, specified as follows:

TIMEpr ¼ b0 þ b1DEVICEr þ b2BROWSERr þb3GRID FLAGpr þ b5NUM WORDSpr þb6SCROLL FLAGpr þ b7SCROLL TIMEpr þ b8DEVICEr�GRID FLAGpr þ mr þ mpr þ epr:

ðQ43Þ

The main effects model explains about 26% of the variation in within-page times, while the interaction model accounts for about 27% of the variation. We can see from Table 6 that device (PC vs. smartphone) remains statistically significant in Model 1, but the sign is negative (faster

Table 4. Percent of Pages With Scrolling by Device and Question Type, 2012 Survey.

Question Type and Scrolling PC (%) Smartphone (%) Overall (%)

All question types: Vertical scrolling 4.28 48.23 8.53 Horizontal scrolling 0.35 9.59 1.24 Any scrolling 4.41 48.95 8.72

Nongrid questions: Vertical scrolling 1.79 31.26 4.15 Horizontal scrolling 0.13 6.60 0.73 Any scrolling 1.83 31.73 4.24

Grid questions: Vertical scrolling 9.75 82.95 16.84 Horizontal scrolling 0.77 15.74 2.22 Any scrolling 10.06 84.11 17.23

Table 5. Least Squares Mean Scrolling and Nonscrolling Times (in Seconds) by Device, 2012 Survey.

Times PC Smartphone

Scroll time for all pages 0.29 4.31 Scroll time for pages with scrolling 5.56 8.37 Nonscroll time (page time minus scroll time) for all pages 14.36 12.47

Couper and Peterson 367

completion times on smartphones) after accounting for other variables. Overall, browser type is only

marginally significant (F ¼ 2.6, df ¼ 4, 4525, p ¼ .035), as we saw earlier. Grid pages take on average almost 8 s longer than nongrid pages. The time taken to respond to a page increases by about

0.2 s per word. This is very close to the typical reading speed for comprehension among college

students, which is about 200 ms per word (see, e.g., Carver, 1992). While the coefficient for the

scrolling flag is negative, we must interpret it together with the coefficient for the amount of time

spent scrolling, which has a significant positive effect on response times.

Turning to the second model in Table 6, we see that the interaction term is statistically significant,

but its inclusion does not add substantially to the proportion of variation explained. To interpret the

interaction of device and grid flags, we look at the least squares adjusted means from Model 2. The

predicted times for grids are 14.4 s for smartphones and 20.1 s for PCs, while for nongrid pages they

are 12.5 and 11.9 s, respectively. This reversal of the predicted time difference for grids reflects the

earlier finding (see Table 5) that when scrolling time (more common in grids, as seen in Table 4) is

excluded (or controlled for in the model), smartphone users appear to take less time to answer than

PC users.

If we remove scrolling time from the measure (i.e., examine nonscrolling time), we get similar

results. The story here is that grid pages appear to take longer to answer than nongrid pages for both

PC and smartphone users. However, for nongrid questions, the within-page time difference is

relatively small once scrolling time is accounted for (which is rare on nongrid items). In other

words, it appears to take just about as long to answer a question on a smartphone as it does on a

PC if scrolling is not necessary.

However, when accounting for the need for scrolling and the amount of time taken to scroll, it

appears that grid questions may actually take longer to answer on PCs than on smartphones. This odd

finding may in part be due to the positively skewed distribution of the dependent variable. However,

examining a model of log times reveals a similar pattern, although less pronounced. We should be

cautious not to overinterpret these model coefficients. But it suggests that the need to scroll—and the

amount of time taken to scroll—accounts for much of the within-page time differences observed

between smartphone and PC users.

Table 6. Nested Main Effects and Interaction Models of Within-Page Time, 2012 Survey.

Variable

Model 1 Model 2

Coefficient SE Coefficient SE

Intercept 3.53 0.12 3.43 0.12 Device (1 ¼ smartphone) �1.04*** 0.28 0.58*** 0.29 Grid (1 ¼ yes) 7.70*** 0.088 8.25*** 0.089 Browser

Chrome — — — — Safari �0.59*** 0.19 �0.71*** 0.19 Firefox �0.25 0.23 �0.24 0.23 IE �0.12 0.31 �0.15 0.31 Other �0.84 0.82 �1.05 0.82

Number of words 0.20*** 0.0017 0.20*** 0.0017 Scroll (1 ¼ yes) �1.31*** 0.13 �0.28* 0.14 Scroll time 1.03*** 0.0086 1.07*** 0.0086 Device*Grid �6.36*** 0.19

Note. IE ¼ Internet Explorer. *p < .05. **p < .01. ***p < .001.

368 Social Science Computer Review 35(3)

We illustrate some of the variation between pages in Table 7. This shows response times for the

three slowest pages (longest overall within-page times) and the three fastest pages. As can be seen,

the three slowest pages all include grids, while the three fastest pages are all single-item pages. The

slowest page (Q15_1) shows the expected difference in mean within-page time by device. But the

second slowest page (Q43), which has a similar level of scrolling for both PC and smartphone users,

does not show much difference in response times. However, the three quickest pages on average

(each containing only a single question and requiring little scrolling on either type of device) still

show some difference in within-page completion times.

It is also noticeable from Table 7 that the between-page response times are relatively stable across

pages (as we discussed earlier) but are somewhat shorter for the faster pages (for both device types).

This suggests that the amount of information being transmitted may account for some of the

between-page time. To explore the between-page time further, we ran a series of multilevel models

predicting between-page time (not shown here). Device remains statistically significant after con-

trolling for other factors, and the adjusted times (0.41 s for PC and 1.01 s for smartphone) remain

very similar to the raw mean times reported earlier. Browser is significantly associated with

between-page times, but the differences are small, ranging from 0.66 s for Chrome to 0.73 s for

Firefox and 0.76 s for other browsers. The number of items on the page and the count of words are

both statistically significant (p < .0001) and positive, suggesting that the amount of information to be

transmitted has some effect on between-page times.

In summary, then, we find that the need to scroll—especially prevalent in grid questions—is a

critical factor in the difference in within-page times between PC and smartphone respondents. When

controlling for the need to scroll and the amount of time taken to scroll, the negative effect of device

on response times is reduced and even (in the case of grid questions) possibly reversed. We turn next

to the 2013 and 2014 surveys before offering a summary discussion of our findings.

2013 Student Survey

Given the similarity of the 2013 survey design to the 2012 survey, we present a brief summary of the

results here, before turning to the 2014 survey. We have data on a total of 3,730 respondents,

including 3,223 completes and 507 partials. This represents a breakoff rate of 13.6%. A total of

Table 7. Examples of Three Slowest and Three Fastest Pages, 2012 Survey.

Page Description Device % Scrolling Between-Page

Time Within-Page

Time

Slowest pages Q15_1 Grid, 7 rows, 5 response options, 94 words PC 12.1 0.44 41.10

SP 90.8 1.26 46.01 Q43 Grid, 4 rows, 6 response options, 106 words PC 10.8 0.41 39.59

SP 86.6 1.12 39.82 Q07 Grid, 11 rows, 4 response options, 78 words PC 79.2 0.45 36.12

SP 98.8 1.19 39.46 Fastest pages

Q61 4 response options, 9 words PC 0.9 0.37 3.40 SP 6.9 0.64 3.79

Q6 5 response options, 12 words PC 0.9 0.36 4.00 SP 8.2 0.77 5.08

Q57 2 response options, 14 words PC 0.6 0.37 4.36 SP 9.8 0.74 6.35

Note. SP ¼ Smartphone.

Couper and Peterson 369

12.7% of respondents (completes and partials) started the survey on a smartphone, up slightly from the 10.8% in 2012 despite the note that the survey was best completed on a PC or tablet. Breakoff rates were significantly higher for smartphones, 27.8% versus 11.5%, w2(1) ¼ 92.3, p < .0001 (see Table 2).

Following extensive cleaning of the paradata, we were left with a total of 176,002 records, nested

within 3,602 respondents, for an average of 48.9 page visits per respondent.

Preliminary analysis of the mean page times shows a mean time of about 15.5 s per page for PC

users and 17.8 s for smartphone users (a difference of about 2.3 s). The within-page mean times are

15.2 and 17.2 s, respectively (a difference of about 2 s), and the between-page mean times are 0.36

and 0.67 s, respectively (a difference of about 0.33 s). This suggests that about 13% of the difference in time is accounted for by between-page time differences (transmission times or network latency),

which is lower than we found for 2012. Overall, the 2013 survey took a little longer per page, but

much of this is due to an increase in within-page (answering) time. Between-page (transmission)

times are basically constant between the two surveys for PC users (0.39 and 0.36 s, respectively), but

lower in 2013 than in 2012 for smartphone users (0.96 and 0.67 s, respectively). This could poten-

tially be attributed to improved network connectivity and speed in 2013.

We replicated the within-page time analyses presented in Table 6 using the 2013 data. These

results (not shown) parallel those found for 2012. The least squares means for the device by grid

interaction is again significant and shows the same pattern as for 2012.

2014 Student Survey

In the 2014 survey, a custom style sheet was deployed that converted all grid items into item-by-item

questions on the same page (what Richards, Powell, Murphy, Yu, & Nguyen, 2015, call a ‘‘stacked’’

approach). This is an increasingly common approach to mobile optimization of grids (see, e.g.,

Lattery, Park Bartolone, & Saunders, 2013; Lorch & Mitchel, 2014; Sarraf, Brooks, Cole, & Wang,

2015), and the new version of the Illume software will include this as a default. Given that all items

were still on the same page, this allows us to examine the effect of grouping items in a grid versus

separating them on page-level response times. Due to a programming error detected after completion

of data collection, this style sheet was also applied to tablets (specifically, iPads). With only 45 iPad

respondents in the sample (out of 3,804 desktop/laptop/tablet users), this is unlikely to have a

substantial effect on the results. We conducted analyses including and excluding these cases, and

the key results did not change, so we retain them in the PC group here.

We have data from a total of 4,935 respondents, of whom 4,233 completed the survey. Smart-

phone users account for 23.8% of those who started the survey, a noticeable increase over prior years. Smartphone users had significantly higher breakoff rates (18.2% vs. 13.0%, w2 ¼ 20.4, df ¼ 1, p < .0001) and were significantly more likely to complete the survey in multiple sessions (23.9% vs. 14.2%, w2 ¼ 60.0, df ¼ 1, p < .0001). After data cleaning and removal of nonquestion pages, our analytic data set contains 238,312 page visits nested within 4,804 respondents, for an average of 49.6

page visit per respondent.

Analysis of the raw mean page times shows a mean time of about 14.5 s per page for PC users and

17.3 s for smartphone users (a difference of about 2.8 s). The within-page mean times are 14.1 and

16.6 s, respectively (a difference of about 2.5 s), and the between-page mean times are 0.37 and 0.68

s, respectively (a difference of about 0.32 s). This suggests that about 13% of the difference in time is accounted for by between-page time differences (transmission times or network latency), which is

very similar to what we found for the 2013 survey but lower than that was found in 2012.

The intraclass correlations from the crossed random effects null models are very similar to those

from 2012 and 2013. For within-page time, 9.0% of the variation is at the respondent level, while 30.4% is at the page level. For between-page time, respondents account for 26.4% of the variation,

370 Social Science Computer Review 35(3)

while pages account for about 0.3%. Again, this suggests that the between-page times are relatively stable across pages.

Table 8 shows the raw (unadjusted) means response times and the means from a nested multilevel

model (including the interaction). As found in previous years, grid pages take significantly longer on

smartphones, even though the grids are converted to a stacked item-by-item format. In fact, in 2012

grid pages take about 7% longer on smartphones than PCs (27.72 vs. 25.93; see Table 3), but the item-by-item pages take on average 20% longer in 2014 (29.40 vs. 24.44; see Table 8). Smartphone users also take significantly longer to complete nongrid pages. The interaction of grid by device is

significant, as we saw before (see also Table 9).

Next, we look at the rate of scrolling. These data are presented in Table 10. The item-by-item

alternative to the grid produced no vertical scrolling for a small subset (about 7%) of pages visited by smartphone users. Further investigation revealed no clear pattern in the devices, operating systems,

or browsers used on these pages, and only 7 smartphone users have no scrolling on any pages they

visited, which suggests that this is not due to technical difficulties in the capture of the scrolling

paradata. It is possible that alternative actions (e.g., pinch and zoom) were used on these pages that

were not captured in the paradata.

The rates of scrolling we see in 2014 are very similar to those in 2012 (see Table 4) and 2013.

While the item-by-item mobile optimization of grids reduced the proportion of pages with horizontal

scrolling, the amount of vertical scrolling increased (as expected). So far, we see similar response

times for grid and nongrid pages for smartphone and PC users as we saw in the previous years and

Table 8. Mean Within-Page Times (in Seconds) by Question Type and Device, 2014 Survey.

Question Type

Unadjusted Means LS Means From Nested Multilevel Model

PC Smartphone PC Smartphone

Grid pages 24.86 29.97 24.44 29.40 Nongrid pages 8.04 9.35 7.72 8.96

Note. LS ¼ least square.

Table 9. Nested Main Effects and Interaction Models of Within-Page Time, 2014 Survey.

Model 1 Model 2

Variable Coefficient SE Coefficient SE

Intercept 7.68 0.097 7.14 0.097 Device (1 ¼ smartphone) �3.07*** 0.19 �0.87*** 0.19 Grid (1 ¼ yes) 14.34*** 0.058 15.87*** 0.062 Browser

Chrome — — — — Safari �0.21 0.16 �0.42** 0.16 Firefox 0.64* 0.28 0.66* 0.28 IE 0.42 1.06 0.26 1.06 Other 0.64 0.33 0.34 0.33

Number of words 0.012*** 0.00048 0.00094*** 0.00048 Scroll (1 ¼ yes) �2.98*** 0.10 �1.74*** 0.10 Scroll time 1.13*** 0.0054 1.25*** 0.0057 Device*Grid �9.22*** 0.15

Note. IE ¼ Internet Explorer. *p < .05. **p < .01. ***p < .001.

Couper and Peterson 371

similar levels of scrolling. We next fit the same nested multilevel models of within-page times for

2014 as we did for 2012 (see Table 6) and 2013. These models are presented in Table 9.

The models in Table 9 are very similar to those for the previous 2 years, suggesting that the

conversion of the grids to a scrolling item-by-item design had little impact on the response time

differential between PC and smartphone users. Scrolling still accounts for much of the time differ-

ence between the two device types.

Looking at the adjusted (least squares) estimates of response time from Model 2 in Table 9, we

again see that, after accounting for scrolling time, the predicted response time is lower for smart-

phone users on grid items than for PC users. One reason for this may be that the scrolling indicator

and time measure in the paradata include the time taken to select a response. This means that the

total scrolling time is likely overstated in the paradata. Nonetheless, the need to scroll and the time

taken to scroll appear to account for a significant proportion of the difference in response times

between PC and smartphone users.

Discussion

As we noted earlier, a number of papers have reported that mobile device users take longer to

complete web surveys than those using PCs. Almost all of those papers have compared overall

survey response times rather than exploring item- or page-level times. Similarly, many of the papers

have offered explanations for the time differences without explicitly examining the reasons for such

observed differences.

This article represents the first attempt to examine variation in time between and within survey

pages by the type of device used. Nonetheless, it suffers from a number of limitations. First, we did

not randomize respondents to device: They self-selected the device used to complete the survey. If

anything, we believe that this may have limited the range of differences that were observed. Those

who choose to use a smartphone to complete the survey are likely more familiar with that device

and would be expected to be more facile (and hence faster) completing a survey using a smart-

phone. Second, the software does not measure the server-side times in milliseconds. This produces

some noise in the estimation of between-page times but is unlikely to alter the general conclusions

we draw. In future research, capturing both server-side and client-side times in milliseconds would

help refine the measurement of between-page times, and we are working with the software vendor

to accomplish this.

Table 10. Percent of Pages With Scrolling by Device and Question Type, 2014 Survey.

Question Type and Scrolling PC (%) Smartphone (%) Overall (%)

All question types: Vertical scrolling 6.86 60.9 18.56 Horizontal scrolling 0.86 7.99 2.41 Any scrolling 7.04 61.45 18.82

Nongrid questions: Vertical scrolling 4.59 40.99 12.43 Horizontal scrolling 0.46 11.86 2.92 Any scrolling 4.64 41.88 12.67

Grid questions: Vertical scrolling 10.52 92.51 28.43 Horizontal scrolling 1.51 1.86 1.58 Any scrolling 10.91 92.53 28.73

372 Social Science Computer Review 35(3)

A third limitation is that we do not have details of the technical capabilities (other than the

browser and operating system) of the devices used. This includes the type of Internet connection

(e.g., 3G, 4G LTE, and Wi-Fi) and the measured speed of the connection. The type of connection

could be ascertained by asking respondents, but they may not be reliable reporters of this informa-

tion. We are currently not aware of any way to passively detect connection type across different

mobile platforms. There are some efforts under way to capture this information (see https://devel-

oper.mozilla.org/en-US/docs/Web/API/NetworkInformation), but this is not yet available across all

browsers and operating systems. Similarly, connection speed can vary depending on server load,

network capacity, and so on, so there is likely considerable variation over time (although our

findings on between-page times suggest these are relatively stable within a single session). This

could potentially be measured unobtrusively by a variety of third-party products (e.g., http://

www.speedtest.net/; https://www.pingdom.com/), but again we are unaware of any successful

implementation in a survey setting.

Finally, we were unable to ascertain whether (and how much) respondents were multitasking,

especially on smartphones. While this may account for some of the difference in time, other studies

(e.g., Antoun, 2015; Mavletova & Couper, 2013) have found that the differences in multitasking

behavior are not as large as might be expected. In other words, both PC and smartphone users do

multitasking while completing the survey. This suggests that multitasking on smartphones may not

account for all of the difference in time but is certainly worth exploring in future studies.

Despite these limitations, we can reach some clear conclusions from our analyses and point to

future research in this area. First, transmission time or connection speed does not appear to

account for all of the time difference that has been observed. We find significant differences in

response times even after restricting the analysis to within-page response times. Further work is

needed in this area, but our analyses (along with those of Mavletova & Couper, 2015) suggest

that a relatively small proportion of the time difference can be attributed to between-page times.

A related conclusion is that the respondent’s choice of browser is not a critical factor in survey

response times. We find small differences in response times across browsers, whether smart-

phone or PC.

It is also clear from the analyses that question type plays an important role in response times. This

finding points to the value of capturing client-side times at the item or page level. The characteristics

of a survey page, including both the number of words and the type of question asked, are important

to measure in comparing response times within and between surveys and devices (see also Couper &

Kreuter, 2013, for interviewer-administered surveys). Grid questions are especially an issue. Several

designers have argued for avoiding grids altogether, and this seems to be particular true for smart-

phones. Mobile optimization for a number of software vendors involves turning grids into single-

item questions for mobile users, but leaving them unchanged for PC users. Our findings suggest that

the ratio of response times for smartphones versus PCs is increased when grids are converted to an

item-by-item layout. The value of this approach, particularly for response times, is yet to be

experimentally tested.

What our analyses seem to suggest that it is not grids per se that are problematic, but the fact that

their size necessitates scrolling, and this is exacerbated on small-screen devices like smartphones.

Converting grids to an item-by-item layout does not reduce the difference in response times. We

believe ours is the first article to measure and report on scrolling behavior in a web survey. Our

analyses clearly implicate scrolling in accounting for response time overall, but especially in

accounting for differences in response times between PC and smartphone users. We are unable to

distinguish between scrolling and other actions (such as pinching and zooming or changing orienta-

tion) that smartphone uses may have taken to navigate large web pages on a small display. It is

possible that these unobserved actions lead to an overestimate of scrolling time and hence an

underestimate of nonscrolling time on smartphones. The scrolling behavior we capture includes the

Couper and Peterson 373

time taken to orient the finger or stylus over the selection. In other words, the paradata does not

identify how long it takes to select a response after scrolling is completed (i.e., the end scrolling time

is the selection time of the response). Further, respondents may well be reading the question and

response options while scrolling. Thus, our results should not be taken to suggest that if scrolling was

eliminated, PC users would take longer than smartphone users.

Nonetheless, while our model coefficients may be biased, the general message seems clear.

Scrolling (both vertical and horizontal) is more prevalent in surveys completed on smartphones

than on PCs. Further, scrolling is more prevalent on grid pages than on nongrid pages. Turning the

grids into item-by-item scrolling pages using a mobile style sheet (as was done in the 2014 survey)

may not result in significant time savings for smartphone users—in fact, we find it increases the ratio

of smartphone to PC time. The fact that grids and multi-item pages require scrolling accounts for a

significant amount of the within-page time differences we observe between smartphones and PCs.

What are the implications of these findings? As noted earlier, several software vendors are

optimizing surveys for mobile devices by converting grids into item-by-item questions (either on

the same page or on separate pages) for mobile devices. Several researchers (see, e.g., de Bruijne

et al., 2015; Peterson et al., 2013; Sarraf et al., 2015; Thomas, Barlas, Graham, & Subias, 2015) are

exploring alternatives to grids for mobile users, including single-item scrolling solutions (with all

items on the same page), single-item paging versions (with each item on a separate page), sliders, or

other alternatives. Our nonexperimental evidence suggests that an item-by-item scrolling approach

does not reduce the time penalty for mobile users. 3

More research is needed to find an optimal

alternative to grids (both in terms of completion time and in terms of data quality) for smartphone

users. For now, we conclude that grids or long multi-item pages should be used with caution if

significant numbers of respondents are likely to use smartphones to complete the survey.

Authors’ Note

The data for this analysis come from 3 years of students surveys conducted as part of the University of

Michigan’s Sustainability Cultural Indicators Program (see http://graham.umich.edu/leadership/scip).

Acknowledgments

We thank the co-PIs, John Callewaert and Robert Marans, for giving us access to the data. We also thank Cheryl

Wiese and Andrew Hupp for their efforts in developing and deploying the surveys, Minako Edgar for her data

management work, and Hueichun Peng and Stuart Downing for technical assistance with the paradata. Patricia

Berglund was very helpful in writing SAS scripts to parse the client-side paradata. We are also grateful to the

reviewers for their helpful suggestions.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publica-

tion of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Notes

1. There are many different forms of mobile optimization. Table 1 includes a binary indicator based on the

descriptions provided in the papers.

2. Based on the research reviewed above, and on our own preliminary analyses, we classify tablet users

together with desktop and laptop users as PC users, while mobile users refer specifically to those using

smartphones. None of our student respondents used feature phones.

374 Social Science Computer Review 35(3)

3. Note that the conversion of grids to item-by-item approaches may be done for reasons other than time—grids

are associated with higher breakoff rates, item missing data rates, and straight-lining or nondifferentiation.

We do not address these issues here.

References

Antoun, C. (2015, May). Effects of mobile versus PC web on survey response quality: A crossover experiment in

a probability web panel. Paper presented at the AAPOR annual conference, Hollywood, FL.

Buskirk, T. D., & Andrus, C. (2012). Smart surveys for smart phones: Exploring various approaches for

conducting online mobile surveys via smartphones. Survey Practice, 5. Retrieved from http://www.survey-

practice.org/index.php/SurveyPractice/article/view/63/pdf

Buskirk, T. D., & Andrus, C. (2014). Making mobile browser surveys smarter: Results from a randomized

experiment comparing online surveys completed via computer or smartphone. Field Methods, 26, 322–342.

Callegaro, M. (2010). Do you know which device your respondent has used to take your online survey? Using

paradata to collect information on device type. Survey Practice, 3. Retrieved from http://www.surveyprac-

tice.org/index.php/SurveyPractice/article/view/250/pdf

Carver, R. P. (1992). Reading rate: Theory, research, and practical implications. Journal of Reading, 36, 84–95.

Chrzan, K., & Saunders, T. (2012, April). Improving the mobile survey experience: A closer look at scale

orientation, grids and modality effects. Maritz Research white paper. Retrieved from http://blog.maritz.de/

wp-content/uploads/2014/02/%E2%80%9EScale-Orientation%E2%80%9C1.pdf

Comer, P., & Saunders, T. (2012, May 30–31). Technical impact of mobile devices. Paper presented at the

CASRO Technology Conference, New York, NY.

Cook, W. A. (2014). Is mobile a reliable platform for survey taking? Defining quality in online surveys from

mobile respondents. Journal of Advertising Research, 54, 141–148.

Couper, M. P., & Kreuter, F. (2013). Using paradata to explore item-level response times in surveys. Journal of

the Royal Statistical Society, Series A, 176, 271–286.

de Bruijne, M., Das, M., van Soest, A., & Wijnant, A. (2015). Presenting grid questions on mobile devices.

Tilburg, the Netherlands: CentERdata working paper, University of Tilburg.

de Bruijne, M., & Wijnant, A. (2013). Comparing survey results obtained via mobile devices and computers:

An experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-

assisted web survey. Social Science Computer Review, 31, 482–504.

de Bruijne, M., & Wijnant, A. (2014). Mobile response in web panels. Social Science Computer Review, 32,

728–742.

Fischer, B., & Bernet, F. (2014, March). Device effects: How different screen sizes affect answer quality in

online questionnaires. Paper presented at the General Online Research conference, Cologne, Germany.

Guidry, K. R. (2012, May). Response quality and demographic characteristics of respondents using a mobile

device on a web-based survey. Paper presented at the annual meeting of the American Association for Public

Opinion Research, Orlando, FL.

Gummer, T., & Roßmann, J. (2014). Explaining interview duration in web surveys: A multilevel approach.

Social Science Computer Review. doi:10.1177/0894439314533479

Heerwegh, D. (2003). Explaining response latencies and changing answers using client-side paradata from a

web survey. Social Science Computer Review, 21, 360–373.

Heerwegh, D. (2011). Internet survey paradata. In M. Das, P. Ester, & L. Kaczmirek (Eds.), Social research and

the internet (pp. 325–348). New York, NY: Taylor and Francis.

Horwitz, R. (2014, October). Usability of the ACS internet instrument on mobile devices. Paper presented at the

Statistics Canada Symposium, Hull, Quebec, Canada.

Hupp, A., Schroeder, H. M., & Piskorowski, A. D. (2014, May). Assessing the impact device choice has on web

survey data collection. Paper presented at the AAPOR annual conference, Anaheim, CA.

Couper and Peterson 375

Jue, A., & Luck, K. (2014). Update: Participation of mobile users in online surveys. Decipher White Paper.

Retrieved November 1, 2014, from https://www.decipherinc.com/n/uploads/images/pages/Decipher_

Mobile_Research_White_Paper_Update.pdf

Kaczmirek, L. (2009). Human-survey interaction: Usability and nonresponse in online surveys. Cologne,

Germany: Herbert von Halem Verlag.

Lambert, A., & Miller, A. (2014). Living with smartphones: Does completion device affect survey responses?

Research in Higher Education. doi:10.1007/s11162-014-9354-7

Lattery, K., Park Bartolone, G., & Saunders, T. (2013, March). Optimizing surveys for smartphones: Maximiz-

ing response rates while minimizing bias. Paper presented at the CASRO Online Research Conference, San

Francisco, CA.

Lorch, J., & Mitchell, N. (2014). Why you need to make your surveys mobile friendly now. Shelton, CT: Survey

Sampling International, webinar slides, downloaded May 24, 2014.

Lugtig, P., & Toepoel, V. (2015). The use of PCs, smartphones, and tablets in a probability-based panel survey:

Effects on survey measurement error. Social Science Computer Review. doi:10.1177/0894439315574248

Lynn, P., & Kaminska, O. (2013). The impact of mobile phones on survey measurement error. Public Opinion

Quarterly, 77, 586–605.

Marans, R. W., & Callewaert, J. (2015). Monitoring the cultural of sustainability at the University of Michigan:

Fall, 2014. Ann Arbor: Graham Sustainability Institute, University of Michigan.

Mavletova, A. (2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31,

725–743.

Mavletova, A., & Couper, M. P. (2013). Sensitive topics in PC web and mobile web surveys: Is there a

difference? Survey Research Methods, 7, 191–205.

Mavletova, A., & Couper, M. P. (2014). Mobile web survey design: Scrolling versus paging, SMS versus

E-mail invitations. Journal of Survey Statistics and Methodology, 2, 498–518.

Mavletova, A., & Couper, M. P. (2015, March). Device use in web surveys: The effect of differential incentives.

Paper presented at the General Online Research conference, Cologne, Germany.

Maxl, E., & Baumgartner, T. (2013, March). Influence of mobile devices in online surveys. Poster presented at

the General Online Research conference, Mannheim, Germany.

McClain, C., Crawford, S. D., & Dugan, J. P. (2012, May). Use of mobile devices to access computer-optimized

web surveys: Implications for respondent behavior and data quality. Paper presented at the AAPOR annual

conference, Orlando, FL.

McClain, C., & Crawford, S.D. (2013, May). Grid formats, data quality, and mobile device use: Toward a

questionnaire design approach. Paper presented at the annual conference of the American Association for

Public Opinion Research, Boston.

McGeeney, K., & Marlar, J. (2013, May). Mobile browser web surveys: Testing response rates, data quality,

and best practices. Paper presented at the AAPOR annual conference, Boston, MA.

Pape, T., & Barron, S. (2013, May). A glimpse inside the mind of a respondent: Using paradata to improve

online surveys. Paper presented at the AAPOR annual conference, Boston, MA.

Peterson, G. (2012, May 30–31). Unintentional mobile respondents. Paper presented at the CASRO Technol-

ogy Conference, New York, NY.

Peterson, G., Mechling, J., LaFrance, J., Swinehart, J., & Ham, G. (2013, March). Solving the unintentional

mobile challenge. Paper presented at the CASRO Online Research Conference, San Francisco, CA.

Richards, A., Powell, R., Murphy, J., Yu, S., & Nguyen, M. (2015, May). Gridlocked: The impact of adapting

survey grids for smartphones. Paper presented at the AAPOR annual conference, Hollywood, FL.

Sarraf, S., Brooks, J., & Cole, J. (2014, May). A look at mobile device usage among college students. Paper

presented at the AAPOR annual conference, Anaheim, CA.

Sarraf, S., Brooks, J., Cole, J., & Wang, X. (2015, May). What is the impact of smartphone optimization on long

surveys? Paper presented at the AAPOR annual conference, Hollywood, FL.

376 Social Science Computer Review 35(3)

Sendelbah, A., Vehovar, V., & Slavec, A. (2014, December). Investigating respondent multitasking in web

surveys: Comparison of self-reports and paradata indicators. Paper presented at the Internet Survey Meth-

ods workshop, Bolzano, Italy.

Sommer, J., Diedenhofen, B., & Musch, J. (2015, March). Not to be considered harmful: Mobile users do not

spoil data quality in web surveys. Poster presented at the General Online Research conference, Cologne,

Germany.

Thomas, R. K., Barlas, F. M., Graham, P., & Subias, T. (2015, February 11-12). What they can’t see can hurt

you: Improving grids for mobile devices. Paper presented at the CASRO Digital Research Conference,

Nashville, TN.

Toepoel, V., & Lugtig, P. (2014). What happens if you offer a mobile option to your web panel? Evidence from

a probability-based panel of internet users. Social Science Computer Review, 32, 544–560.

Wells, T., Bailey, J., & Link, M. (2014). Comparison of smartphone and online computer survey administration.

Social Science Computer Review, 32, 238–255.

Yan, T., & Tourangeau, R. (2008). Fast times and easy questions: The effects of age, experience and question

complexity on web survey response times. Applied Cognitive Psychology, 22, 51–68.

Zahariev, M., Ferneyhough, C., & Ryan, C. (2009). Best practices in mobile research. Proceedings of the

ESOMAR Online Research Conference, Chicago, IL, ESOMAR, 164–176.

Zwarun, L., & Hall, A. (2014). What’s going on? Age, distraction, and multitasking during online survey

taking. Computers in Human Behavior, 41, 236–244.

Author Biographies

Mick P. Couper is a research professor in the Survey Research Center, University of Michigan, and the Joint

Program in Survey Methodology. E-mail: [email protected]

Gregg J. Peterson is the associate director of Survey Research Operations in the Survey Research Center,

University of Michigan. E-mail: [email protected]

Couper and Peterson 377