RCH Unit III

63

Journal of MultiDisciplinary Evaluation

Volume 13, Issue 29, 2017

ISSN 1556-8180

http://www.jmde.com

Examining Factors Impacting Online Survey Response Rates in Educational Research: Perceptions of Graduate Students

Amany Saleh Arkansas State University Krishna Bista Morgan State University Background: In educational research, online survey has become one of the most popular methods of data collection. Academic researchers, including faculty and students, expect and require a good response rate to their research projects for reliable results. Purpose: In this paper, the authors examine a wide range of factors related to survey response rates in academic research. Examples include email checking habits, survey design, and attitudes toward research. Setting: An online survey environment Intervention: Not applicable. Research Design: A cross-sectional quantitative research method was used to analyze the factors that influence participants’ email survey response rate. Data were collected at a single point in time. The authors did not directly measure changes that come over time in this study.

Data Collection and Analysis: After receiving the Institutional Research Board’s approval, the researchers distributed the survey via the American Educational Research Association (AERA) Graduate Student Discussion List subscribers. A sample of 454 responses was used in the final analysis-- with a 78.9 % response rate. The authors used descriptive statistics (percentage, average mean) and inferential statistics (chi- square and correlations) to report the data analysis and findings. Findings: Results indicated that research survey response rate was highly influenced by interests of participants, survey structure, communication methods, and assurance of privacy and confidentiality. The findings also suggested that male participants were more likely to respond to surveys if they received a reminder, and older participants were more likely to respond if they were promised a reward.

Keywords: academic research; education graduate students; online surveys; postal surveys; survey response rates

64

Figure 1. Factors influencing survey response rates

Introduction

In educational research, the online survey has become a popular method of data collection. Academic researchers including faculty and students are more prone to use online surveys due to quick responses. Utilizing surveys, a favorable response rate is required to obtain satisfactory results for the research project. Online surveys have many advantages as well as disadvantages. One advantage scholars have reported is that the online surveys yield significantly higher response rates than the paper surveys (Koundinya, Klink, Deming, Meyers, & Erb, 2016; Liu & Wronski, 2017; Schonlau, Fricker, & Elliott, 2002). One of the major issues with online surveys is having an updated and accurate email address list for potential participants. Based on the authors’ experiences, many people active in education tend to have more than one email addresses, often times including an email address that may be rarely checked. Silva and Durante (2014) reasoned that a majority of people use the Internet for entertainment and recreation, which makes them discount research survey participation requests, resulting in a low response rate. Still, Silva and Durante (2014) asserted that the Internet holds great potential for future research. In the last three decades, online surveys have become the predominant method of eliciting participation in academic research for its ease, quick response, and low cost. Educational scholars have noticed a decline in the response rate of online surveys compared to postal surveys (Fan & Yan, 2010; Fosnacht, Sarraf, Howe, & Peck, 2017; Roberts & Allen, 2015; Shannon & Bradshaw,

2002; Sheehan, 2001). Based on our personal experience of supervising doctoral students in their graduate research projects, students are not willing to use postal mail for its cost compared to e-mail surveys. Some survey platforms such as Qualtrics and Survey Monkey allow researchers to easily transfer the data to sophisticated statistical analysis programs. Online surveys also record the number of deliverable e-mails keeping an accurate measure of the sample size. While, others have a feature to track the respondents’ IP addresses, so they do not send follow up reminders to those who already completed the survey. Still, we have witnessed a remarkable decrease in the response rate of e-mail surveys in the last decade. A low response rate of online surveys has been a concern for many researchers in the last few years; the response rate for web surveys is estimated to be 11% lower than other survey modes (Yan & Fan, 2010). In this study, we examined the factors that influence education graduate students’ responses to online surveys such as email checking habits, survey design, and attitudes toward research.

Theoretical Framework

Dillman’s (2007) social exchange theory contends that humans’ behavior is motivated by the response they expect to elicit. In other words, survey response rates depend on a combination of factors including reward, trust, and costs. All of these factors affect the likelihood that individuals will respond to surveys. Elaborating on the three elements that proffered, this study hypothesized that ease and low cost (represented in the time

Journal of MultiDisciplinary Evaluation

65

taken to fill the survey by the respondents); origin or sponsor of the survey, structure and length of the survey, assurance of confidentiality, and the use of pre-notification and reminders (all portraying professionalism on the part of the researcher); and the use of rewards and/or incentives as well as interest in the focus of the research will be identified as influential factors in increasing web survey response rates. Literature Review

Online surveys have many advantages over traditional postal surveys, including a reduction in cost and ease of analysis (Andrews, Nonnecke, & Preece, 2003; McPeake, Bateson, & O'Neill, 2014). For example, Sheehan and McMillan (1999) compared e-mail to mail surveys and found that e- mail surveys took an average of 7.6 days while mail surveys took an average of 11. 8 days to return. The cost of e-mail surveys they estimated to be 5-12% of the cost of mail surveys. In another study among college faculty members, Shannon and Bradshaw (2002) found that electronic surveys are cheaper and faster than postal surveys but the response rate was not as high as that generated by mail surveys. They also argued that online surveys have drawbacks such as selection bias and poor response rates. Uhlig, Seitz, Eter, Promesberger, and Busse (2014) found that Internet-based surveys are time and cost efficient for populations larger than 300. On the other hand, paper or mail surveys can be more appropriate for smaller populations. Online surveys also have distinctive features related to design, distribution and evaluation of data. The relevance of the topic to the responders is also a major factor in the response rate for any survey (Dillman, 2007). The length of survey is another significant factor in response rate (Liu & Wronski, 2017; Porter, 2004b). Based on 25,080 real-world web surveys, Liu and Wronski (2017) found a negative relationship between completion rate and survey length and question difficulty. They also suggested that surveys without progress bars have higher completion rates than surveys with progress bars. McPeake et al. (2014) suggested keeping the survey as short as possible. Two early studies found that surveys which take thirteen minutes or less have higher response rates (Asiu, Antone, & Fultz, 1998; Handwreck, Carson, & Blackwell, 2000). How the surveys are designed and presented to the target population is another factor that influences the response rate (Crawford, McCabe, & Pope, 2005; Dillman, & Smyth, 2007). Fan and Yan (2010) established five survey

delivery elements affecting survey response rate. These elements are: sample; delivery mode; invitation design; the use of pre-notification and reminders; and the use of incentives. Targeting a population that has access to the Internet and an interest in the survey topic is vital. Student and employee populations are more likely to respond to surveys than the general population (Shih & Fan, 2008; Vance, 2011). Yet, another study showed that students, especially at public schools, have low response rate to surveys (Perkins, 2011). Similarly, Margo, Prybtok and Ryan (2015) found a significant difference in response rates between participants who were offered extra credit and those who were not. In academic research, personalization of invitations using personal greetings, titles, and addresses significantly increased response rate (Heerwegh, Vanhove, Matthijs, & Loosveldt, 2003; Joinson, Woodley, & Reips, 2007). Asking for help from responders is influential in increasing response rate (Groves, Cialdini, & Couper, 1992; Porter, 2004b). Petrovčič, Petrič, and Lozar Manfreda (2016) also found a plea for help was effective for eliciting responses in email invitations. Porter (2004b) found that people were more likely to respond to survey requests from authority figures or if they were addressed as a part of a select group chosen to fill out the survey (2004a). Including a deadline and a statement of the selectivity of the responders in the survey invitation were found to be statistically significant in improving survey response rates (Porter, & Whitcomb, 2005). Personalizing the invitation to participate in the survey and frequent reminders significantly impacted the response rate as Muňoz- Leiva et al. reported from their study (2010). However, one study based on 1,598 selected students indicated that neither the degree of personalization nor the length of the invitation email impacted survey response or completion (Trespalacios & Perkins, 2016). The use of pre-notification and reminders was demonstrated to be another significant element in survey response rate (Bosnjak, Neubarth, Bandille, & Kaczmire, 2008; Fox, Schwartz, & Hart, 2006; Kaplowitz, Hadlock, & Levine, 2004; Porter, 2004b; Spruyt & Van Droogenbroeck, 2014; Trouteaud, 2004; Veen, Göritz, & Sattler, 2016). For an example in a recent study, the response rate increased from 22.6% to 39.4% in their study after the reminders were sent (Aerny-Perreten, Domínguez-Berjón, Esteban-Vasallo, & García- Riolobos, 2015). Yet, scholars have advised against sending more than three to four reminders (Muňoz-Leiva, Sánchez-Fernández, Montoro-Ríos, & Ibáňez-Zapata, 2010; Van Mol, 2017). Based on

Saleh & Bista

66

a sample of 15,651 students in Belgium, Van Mol’s (2017) study found that the effects of over- surveying have resulted in a decline in the overall response rate. Research has also found that pre-paid incentives increased response rates (Mercer, Caporaso, Cantor, & Townsend, 2015; Porter, 2004a) while post-paid incentives had no influence on response rates (Goritz, 2006; Porter, & Whitcomb, 2003). The amount of incentives, however, did not show a clear, significant effect on survey response rates (Bosnjak, & Tutem, 2003; Dykema et al., 2012; Goritz, 2006; Porter, & Whitcomb, 2003). Results of another study indicated that a mixed-mode survey which involves mail and email components has produced moderately high response rates. The study also found that the mail survey was more effective than the email survey (Converse, Wolfe, Huang, & Oswald (2008). Fan and Yan (2010) asserted that sponsors of the survey can affect response rates. They maintained that surveys sponsored by academic and governmental agencies have higher response rate than commercially sponsored surveys, an assertion supported by Vance (2011). One study indicated that contextual cues (e.g., researcher identity, sponsor identity and incentive types) also affect response rate of online surveys (Pan, Woodside, & Meng, 2013). In their study, a high- power position (identity) of the researcher generated more responses than one in a lower power position (Pan et al, 2013). Magro, Prybutok and Ryan (2015) found higher responses among student participants using an extra credit approach in electronic surveys. In other student participant research, Ravert, Gomez-Scott and Donnellan (2015) used iPads for an alternative electronic survey. Their study did not find any significant differences between paper and iPad- based survey in terms of acceptance rates and proportion of missing data.

Methods Research Questions

This study was guided by the following research questions:

1. To what extent do factors such as email checking habits, attitude towards the research, rewards, length of the survey, the survey structure, assurance of privacy,

and frequency of reminder influence survey response rates?

2. Are demographic factors such as gender or age related to participants’ responses to survey items?

Research Design

A cross-sectional quantitative research method was used to analyze the factors that influence participants’ email survey response rate. In cross- sectional research, data were collected from the research participants at a single point in time or relatively short time period (Johnson & Christensen, 2017). Thus, the researchers did not directly measure changes that come over time in this study but rather used descriptive statistics (percentage, average mean) and inferential statistics (chi-square and correlations) to report the data analysis and findings.

Participants The participants of this study were graduate students in education or related fields. Participants were either masters or doctoral students who were American Educational Research Association (AERA) graduate student discussion listserv members. AERA (2017) estimates that 30% (about 7,500) of their 25,000 members are graduate students. This student population was selected based on their active engagement in the education research association. Furthermore, we also anticipated that graduate students will be interested in participating in the study because they may be using or considering survey methodology in their research papers and dissertations. After gaining the approval of Institutional Research Board (IRB), the participants were invited to take part in the study using Survey Monkey, an online survey collection tool. We distributed the survey via the listserv. The survey took approximately five to seven minutes to complete. The participants also received a reminder two weeks after the initial survey distribution. Data were collected two weeks after sending the reminder. The total population of the listserv was 575 AERA Graduate Student Discussion List subscribers. Statisticians (Gal, Gal, & Borg, 2007; Krjcie & Morgan, 1970) recommend a sample of 366 for a population of 7,500. With 454 responses, our sample exceeded this standard with a 78.9 % response rate.

Journal of MultiDisciplinary Evaluation

67

Table 1 General Perceptions of Participants Related to Survey Response Factors (N = 454)

Category Item Agree Disagree Neutral Email Checking Habit

I open all my emails 40.4% (183)

49.9% (226)

9.7% (44)

I only open emails from people I know 51.6% (233)

14.2% (64)

34.2% (155)

I am more likely to open an email from an organization I belong to

91.2% (413)

5.7% (26)

3.1% (14)

I am more likely to open an email from a non-profit organization

20.4% (93)

46.9% (213)

32.6% (148)

Attitude Toward Research

I am more inclined to complete a research conducted by a student

51.5% (215)

37.5% (170)

15% (68)

I am more inclined to complete a research survey if it was forwarded by my Chair

74.1% (334)

17.1% (77)

8.9% (40)

I am more inclined to answer the survey if it is from a colleague I know

88.7% (399)

8% (36)

3.3% (15)

I am more inclined to answer the survey if it is from a colleague in the profession whom I do not know personally

39.2% (177)

34.5% (156)

26.3% (119)

Interest I am more inclined to complete the survey if I have a vested interest in the topic

88.2% (389)

8.4% (37)

2.9% (13)

I am more inclined to complete the survey if the author promised to share the results with me

49% (256)

32% (141)

19.1% (84)

I am more likely to fill out academic (educational) surveys 76.3% (338)

18.3% (81)

5.4% (24)

Rewards I am more inclined to complete the survey if promised a monetary reward

38.4% (169)

27% (119)

34.7% (153)

I am more inclined to complete the survey if I received a reward with the survey invitation

37.9% (168)

27.5% (122)

34.6% (153)

Length of Survey I am more inclined to complete the survey if I know how long it will take to fill out beforehand

87.1% (384)

9.3% (41)

3.6% (16)

I am more inclined to complete the survey if it takes less than 15 minutes

91.1% (391)

6.8% (30)

2% (9)

Value Privacy I am more inclined to answer the survey if I am assured of anonymity

75% (233)

18.9% (83)

5% (22)

I am more inclined to complete the survey if I am assured that my answers will remain confidential

78.3% (343)

17.6% (77)

4.1% (18)

Survey Structure I am more likely to open an email with subjects clearly indicating the research nature of the content

87.4% (377)

7.7% (35)

4.8% (22)

I am more likely to fill out the survey if the email invitation looked professional

85.1% (373)

11.6% (51)

3.2% (14)

I am more likely to complete the survey if the questions items are short and concise

94.1% (410)

4.6% (20)

1.3% (6)

I am less likely to finish the survey if it included open- ended questions

57% (248)

23.9% (104)

19.1% (83)

I am more likely to fill out the survey if the email invitation includes my name

41.1% (180)

40.3% (176)

18.5% (81)

Saleh & Bista

68

Reminder I am more likely to complete the survey if I receive a reminder

66.6% (289)

20.8% (91)

12.6% (55)

I get bothered if I received more than three reminders from the researchers.

75.3% (328)

16.5% (72)

8.3% (36)

Pre-notification I am more likely to open the email if I received a pre-notification

Response Percent

Count

a) By email 91.8% 370 b) By mail 8.2% 33

Survey Time Received

I am more inclined to complete the survey if I received the email Response Percent

Count

a) At beginning of the day 53.2% 223 b) During my lunch/break time 13.6% 57 c) At the end of the day 13.4% 56 d) In the evening 19.8% 83

I am less likely to answer the survey Response Percentage

Count

a) During the summer months 14.5% 63 b) During the holidays 39% 164 c) At the beginning of the school year 20% 84 d) Toward the end of the school year 26.6% 112

Survey Instrument We developed the survey instrument based on existing literature (Dillman, 2007; Fan & Yan, 2010; Vance, 2011). The instrument included 28 items of which there were 24 items related to electronic survey response (Cronbach’s Alpha = .766) and two demographic variables. With the exception of the questions eliciting demographic information, the instrument utilized a Likert scale where 1 = strongly agree and 5 = strongly disagree. In reporting the results, the data from the two columns of “strongly agree” and “agree” were combined and the data for “strongly disagree” and “disagree” were combined as well. The survey instrument included the following sub- group of variables: email checking habit (4 items), attitude toward research (4 items), similar research interests (3 items), rewards for completion (2 items), length of survey (2 items), assurance of privacy and confidentiality (2 items), survey structure (5 items), survey reminders (2 items), time survey was received (2 items) and demographic information (2 items). Results A total of 454 participants completed the online survey. There were more female (72.4%, 314) than male student participants (27.6%, 120). Twenty students did not respond to the gender question.

About half of the total student participants were in the age group of 31-45 years (44.1%), whereas a quarter of participants were either younger than 30 years or between 46 to 60 years. There were 31 (7.1%) participants who identified themselves as older than 60 years. Participants reported a variety of email checking habits (Table 1). The majority of participants reported that they only open emails from the people they know or emails from the organization for which they work. Almost half the participants (49.9%) reported that they do not open all emails they receive. Participants also showed a higher percentage of positive response (51.5% or more) to the questions related to their attitude toward research. Participants indicated that they were willing to complete a research survey conducted by a student (51.5%), received from the department chair (74.1%) or sent by known colleagues (88.7%). About 40% of participants reported that they were willing to complete the survey if it was sent from a colleague whom they do not know personally. Results also indicated that the survey response rate was highly related to research interests of participants, rewards associated with survey completion,; length of survey; and assurance of privacy and confidentiality. When asked, 88.2% of participants agreed that they were more inclined to complete the survey if they have a vested interest in the topic; 76.3% of participants indicated that they will complete a survey if it was

Journal of MultiDisciplinary Evaluation

69

educational or academic in nature. About 91.1% participants agreed that they would complete the survey if it takes less than 15 minutes to complete. More than three quarters of participants (78.3%) agreed that the researchers should assure them of the anonymity and confidentiality of the information they share in the survey (see Table 1). Only 38% of participants indicated that a monetary reward was expected when they participate in the survey. Results indicated three important factors necessary in order to receive a high response rate. These factors were: pre-notification sent by email (91.8%); survey structure -an email with a clear research subject heading (87.4%); professional email invitation (85.1%); short and concise question items (94.1%); few or no open-ended questions (57%); and a reminder (66.6%). Participants were asked for the time of day they were more likely to complete the survey. Over 53% of them replied that they would complete it if received at the beginning of the day. They also indicated that they were less likely to complete a survey if they received it during holidays (39%), summer (14.5%), beginning of the school year

(20%), or toward the end of the school year (26.6%). Pearson Chi-square results indicated that male participants were significantly different from females on whether they were likely to complete the survey if they received a reminder (X2 = 7.86, df = 2, N = 454, p = .020), and if the questions/statements were short and concise (X2 = 7.68, df = 2, N = 454, p = .022). In other words, males were more likely to complete the survey after a reminder than females. Similarly, males were more likely than females to complete the survey if the questions were short and precise. However, males and females were not significantly different on whether they were likely to open the survey email if they received a pre-notification by email or by mail (X2 = .002, df = 1, N = 454, p = .964). Phi, which indicated the strength of the association between two variables, was -.002. Similarly, male and female participants were not significantly different on whether they would complete a survey if they had a vested interest in the research topic (X2 = .257, df = 2, N = 454, p = .879). The strength of the association between variables had a smaller size effect of .135 or less.

Table 2. Chi-Square Analysis of Prevalence of Survey Response Factors Among Males and Females

Variable N Gender

X2 P Male Female

I am more likely to open the survey email if I received a pre-notification .002 .964

By email 364 101 263 By mail 32 9 23

I am more likely to complete the survey if I receive a reminder 7.86 .020

Agree 289 71 218 Neutral 89 25 64

Disagree 56 24 32 I am more likely to complete the survey if the question items are short and concise

7.68 .022

Agree 406 106 300 Neutral 19 10 9

Disagree 9 4 5 I am inclined to complete the survey if I had a vested interest in the topic .257 .879

Agree 378 103 275 Neutral 37 11 26

Disagree 19 6 13

70

Our correlational analysis indicated that the age of participants was positively correlated with rewards, r (435) = .22, p < .001. However, age was not statistically correlated with participants' email checking habits, attitude toward research, length of survey, privacy assurance, survey structure, and the number of reminders. Further analysis indicated that younger participants, age 20-30, were less interested in rewards than older participants, age 40-60 or older.

Discussion

Even though e-mail contact is the fastest, lowest- cost method to deliver surveys, the use of anti- spamming software and the wide use of unsolicited email made the response rate of email surveys lower than mail or phone surveys (Couper, Kapteyn, Schonlau, & Winter, 2007; Petrovčič et al., 2016; Vance, 2011; Veen et al., 2016). In the 1990s, email surveys were a promising method to gain a better response rate on research surveys. Muňoz-Leiva, et al. (2010) reported that the response rate for email surveys was above 50% in the early 1990s due to the novelty of email, but now this percentage had declined greatly due to the wide use of filters and survey fatigue. Sheehan (2001) pointed out that research on email surveys offer little information on research design and methodology. Porter (2004b) stated that it is vital for researchers to understand the reasons people respond to surveys. The results of this study suggest that participants prefer completing electronic surveys received mostly from students, colleagues and authority figures (e.g. department chair or higher) compared to people from other organizations who they do not know personally or professionally. These results support earlier findings by Fan and Yan (2010), Pan et al. (2013), Porter (2004b) and Trespalacios & Perkins (2016). One of the implications of these results may be that researchers should target specific groups or elicit the aid of authority figures in the field of the target population, for example, a department chair or CEO of an organization. Researchers can also enlist the help of major organizations to distribute the surveys through their listserv. Results also indicated that the survey response rate was highly related to the research interests of participants. The results are in agreement with the findings of Dillman (2007) and Porter and Whitcomb (2005). Over 88% of participants indicated that they will be more likely to complete a survey if they are interested in the topic. Therefore, researchers need to target populations

that are more likely to be interested in the research focus to increase the response rate for their surveys (Petrovčič et al., 2016). In this study, we targeted graduate students involved in educational research assuming that they will be interested in the results. This could be the reason that the response rate for this survey was 78.9%. Past studies asserted that shorter surveys have higher response rate (Liu & Wronski, 2017; Porter, 2004a). Similarly, about 91.1% participants in this study agreed that they would complete the survey if it takes less than 15 minutes to complete. Researchers need to be mindful of the time it takes to complete their survey and attempt to shorten their survey, when possible, to motivate respondents to complete it. There is also a need to inform the participants in the invitation letter of the approximate time it will take to complete the survey. In the current study, we informed the participants that it will take approximately 5-7 minutes to complete. More than three quarters of participants (78.3%) reported that the study should assure anonymity and confidentiality for the information they share with researchers in the survey. The results of the study indicated important factors necessary to receive a high response rate: pre- notification sent by email (91.8%); survey structure (an email with a clear research subject heading (87.4%); professional email invitation (85.1%); short and concise question items (94.1%); and few or no open-ended questions (57%); and a reminder (66.6%). These findings supported earlier findings (Aerny-Perreten et al., 2015; Crawford, McCabe, & Pope, 2005; Dillman, & Smyth, 2007; Greer, Chuchinprakarn, & Seshadri, 2000; Spruyt & Van Droogenbroeck, 2014; Van Mol, 2017). Therefore, it is suggested for researchers to consider these factors as they design their online surveys and research methods. Greer, Chuchinprakarn, and Seshadri (2000) found the day of the week the survey was sent not to be an influential factor in response rate. In this study, over 53% of the respondents indicated that they are more likely to complete the survey if they received it at the beginning of the day. The results of this study indicated that the time of the year may have an effect on the response rate for surveys. Almost 40 % of participants agreed that they are less likely to answer the survey during the holidays. Others indicated that they are less likely to answer the survey during the summer months (14.5%), at the beginning (20%) or end of the school year (26.6%). The results are in agreement with Vance’s (2011) contention that seasonality affects survey response rate and that it differs with each target group. The results of this study reflect

Journal of MultiDisciplinary Evaluation

71

the times where graduate students are either on vacation or involved in their studies. These results suggest that future researchers have to be cognizant of the effect of timing on research survey response rates. Researchers should choose a time during the year that is most conducive for their target population. Additionally, male participants were significantly different from females in other parts of the study. Males were more likely to complete the survey if they received a reminder, and if the questions items were short and concise. Results also indicated that rewards offered in the survey correlate positively with age. In other words, the older the participants, the higher the response rate if a reward is promised. The results agree with the findings of previous studies (Couper, Conrad, & Tourangeau, 2007; Yan & Fan, 2010) that some demographic variables such as age and race affect participants’ willingness to complete the surveys. We contend the survey response rate depends on a combination of factors including reward, trust and costs, which affect the likelihood that individuals will respond to the surveys. This contention is supported by Dillman’s (2007) social exchange theory. He argued that human behavior is motivated by the response they expect to elicit. A major limitation to the study is that only graduate students of education were included. This group of students may not represent the general population of education graduate students because they are listed on the AERA listserv, which indicate that they are more involved in educational research than others. Therefore, the results cannot be generalized for the whole population of graduate students of education, nor it can be generalized for the general population. Recommendations This study offers some insights regarding the use of online surveys and response rates. It may prevent many researchers from becoming disappointed from the dismal response rates for their surveys. As Porter and Whitcomb (2005) described, low response rate in online survey research is an increasing phenomenon, which needs to be addressed. In the current study, we examined the factors that influence education graduate students’ responses to online surveys. Based on responses of participants in this study, we suggest that future researchers may consider the following practices when designing their surveys and invitation letters to research target populations in order to increase their response rates. Researchers are recommended to :

1. Elicit the aid of authority figures, known personnel or organizations to the target population to distribute the survey, when possible.

2. Target a population that is more likely to hold interest in the research.

3. Consider offering an incentive for completing the survey.

4. Make every effort to craft a survey that is short and concise.

5. Inform the population in the invitation letter of the approximate time it will take to complete the survey.

6. Whenever possible, reduce the number or eliminate open-ended survey items.

7. Assure the participants of the anonymity and confidentiality of their responses.

8. Explain how the collected data will be handled, who will have access to them, and how the data will be stored and/or disposed of after the study is completed.

9. Personalize invitations to participate in the study and make them look professional.

10. Send at least one, but not more than three, reminders to the target population to motivate them to complete the survey.

11. Be aware of the time constraints related to time-of-year for the target population.

This study offers insights as to why education graduate students were interested in responding or not responding to online surveys. In the case of this study, graduate students in the field of education served as a purposeful, albeit narrow, sample. However, this sample is not free from limitations. One limitation is that the graduate student listserv is opt-in for graduate students, meaning that only a particular subsection of AERA-affiliated graduate students will receive emails sent to that listserv. Instead, these inferences are really about how those who respond to research invitations rated their own likelihood to respond. We suggest the future research could focus on reasons why many research survey invitations remain unanswered. Similar research needs to target different populations with different interests, fields or occupations as well as other countries. Future research should examine other demographic elements such as race, ethnicity, degree of education, and personality of the participants. The model and recommendations may help future researchers, but they should not be generalized to all populations until similar studies are conducted with different populations and findings of such studies are supported.

Saleh & Bista

72

References

American Educational Research Association (AERA) (2017). Who we are. Retrieved from

http://www.aera.net/About-AERA/Who-We- Are

Aerny-Perreten, N., Domínguez-Berjón, M. F., Esteban-Vasallo, M. D., & García-Riolobos, C.

(2015). Participation and factors associated with late or non-response to an online survey in primary care. Journal of Evaluation in Clinical Practice, 21(4), 688-693.

Andrews, D., Nonnecke, B., & Preece, J. (2003). Electronic survey methodology: A case study

in reaching hard-to-involve internet users. International Journal of Human- Computer Interaction, 16(2), 185.

Asiu, B. W., Antone, C. M., & Fultz, M. I. (1998), Undergraduate perceptions of survey

participation: Improving response rates and validity. A paper presented at the Annual Meeting of the Institutional Research, Minnesota.

Bosnjak, M., Neubarth, W., Couper, M.P., Bandille, W., & Kaczmire, L. (2008). Pre- notification in web-based access panel surveys - The influence of mobile text messaging versus email on response rates and sample composition. Social Science Computer Review, 26, 213-223.

Bosnjak, M., & Tuten, T. L. (2003). Prepaid and promised incentives in web surveys. Social Science Computer Review, 21, 208-217.

Converse, P. D., Wolfe, E. W., Huang, X., & Oswald, F. L. (2008). Response rates for mixed-mode surveys using mail and e- mail/web. American Journal of Evaluation, 29(1), 99-107.

Couper, M. P., Conrad, F. G., & Tourangeau, R. (2007). Visual context effects in web surveys. Public Opinion Quarterly, 71, 623-634.

Couper,M. P., Kapteyn, A., Schonlau, M., & Winter, J. (2007). Non-coverage and non- response

in an internet survey. Social Science Research, 36, 131-148.

Crawford, S. D., McCabe, S. E., & Pope, D. (2005). Applying web-based survey design standards. Journal of Prevention and Intervention in the Community, 29, 43-66.

Dillman, D. A. (2007). Mail and Internet surveys: The total design method for surveys. New York, NY: Wiley.

Dillman, D. A., & Smyth, J. D. (2007). Design effects in the transition to web-based surveys.

American Journal of Preventive Medicine, 32(55), 90-96.

Dykema, J., Stevenson, J., Klein, L., Kim, Y., & Day, B. (2012). Effects of e-mailed versus mailed invitations and incentives on response rates, data quality, and costs in web survey of university faculty. Social Science Computer Review, 31(3), 359-370. doi: 10.1177/089443931246254

Fan, W., & Yan, Z. (2010). Factors affecting response rates of the web surveys: A systematic review. Computers in Human Behavior, 26, 132-139. doi: 10.1016/j.chb.2009.10.015

Fosnacht, K., Sarraf, S., Howe, E., & Peck, L. K. (2017). How important are high response rates for college surveys? Review of Higher Education, 40(2), 245-265. doi:10.1353/rhe.2017.0003

Fox, G., Schwartz, A., & Hart, K. M. (2006). Work- family balance and academic advancement in medical schools. Academic Psychiatry, 30, 227-234.

Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An introduction. New York, NY: Pearson.

Goritz, A. S. (2006). Incentives in web studies: Methodological issues and a review. International Journal of Internet Science, 1, 58-70.

Greer, T. V., Chuchinprakan, N., & Seshadri, S. (2000). Likelihoods of participating in mail survey research: Business respondents’ perspectives. Industrial Marketing Management, 29, 97-109.

Groves, R. M., Cialdini, R. B., & Couper, M. P. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly, 56, 475-495.

Handwerk, P., Carson, C., & Blackwell, K. (2000). Online versus paper-and-pencil surveying of students: A case study. A paper presented at the Annual Association of Institutional Research Conference, Cincinnati, Ohio.

Heerwegh, D., Vanhove, T., Matthijs, K., & Loosveldt, G. (2003). The effect of personalization on response rates and data quality in web surveys. International Public Opinion Quarterly, 40, 374-375.

Joinson, A. N., Woodley, A., & Reips, U.D. (2007). Personalization, authentication, and self disclosure in self-administered Internet surveys. Computers in Human Behavior, 23, 275-285.

Johnson, R. B., & Christensen, L. (2017). Educational research: Quantitative,

Journal of MultiDisciplinary Evaluation

73

qualitative, and mixed approaches (6th ed.). Thousand Oaks, CA: Sage.

Kaplowitz, M. D., Hadlock, T. D., & Levine, R. (2004). A comparison of web and mail survey response rates. Public opinion Quarterly, 68 (1), 94-101.

Koundinya, V., Klink, J., Deming, P., Meyers, A., & Erb, K., (2016). How do mode and timing of follow-up surveys affect evaluation success? Journal of Extension, 54(1). Retrieved from https://www.joe.org/joe/2016february/rb1.ph p

Krejcie, R. V., & Morgan, D. W. (1970). Determining sample size for research activities. Educational and Psychological Measurement, 30, 607-610.

Liu, M., & Wronski, L. (2017). Examining completion rates in web surveys via over 25,000 real-world surveys. Social Science Computer Review. Retreived from http://journals.sagepub.com/doi/abs/10.1177 /0894439317695581

Magro, M. J., Prybutok, V. R., & Ryan, S. D. (2015). How survey administration can affect response in electronic Surveys. Quality & Quantity: International Journal of Methodology, 49(5), 2145-2154. doi:10.1007/s11135-014-0098-4

McPeake, J., Bateson, M., & O'Neill, A. (2014). Electronic surveys: how to maximise success. Nurse Researcher, 21(3), 24-26. doi:10.7748/nr2014.01.21.3.24.e1205

Mercer, A., Caporaso, A, Cantor, D., & Townsend, R. (2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79(1), 105-129. doi:10.1093/poq/nfu059

Muňoz-Leiva, F., Sánchez-Fernández, J., Montoro- Ríos, F., & Ibáňez-Zapata, J. A. (2010). Improving the response rate and quality in web-based surveys through the personalization and frequency of reminder mailings. Qual Quant, 44. 1037-1052.

Pan, B., Woodside, A. G., & Meng, F. (2013). How contextual cues impact responses and conversion rates of online surveys. Journal of Travel Research, 53(1), 58-68. doi: 10.1177/0047287513484195

Perkins, R. A. (2011). Using research-based practices to increase response rates of web- based surveys. Educause Review Online. Available at http://www.educause.edu/ero/article/using- research-based-practices-increase-response- rates-web-based-surveys

Porter, S. R. (2004a). Raising response rates: What works? New Directions for Institutional

Research, 121, 5-21. Porter, S. R. (2004b). Overcoming survey research

problems: New directions for institutional research. London: Jossey-Bass.

Porter, S. R., & Whitcomb, M. E. (2005). Non- response in student surveys: The role of demographic engagement and personality. Research in Higher Education, 46 (2), 127-152.

Porter, S. R., & Whitcomb, M. E. (2003). The impact of lottery incentives on student survey response rates. Research in Higher Education, 44, 389-407.

Petrovčič, A., Petrič, G., and Lozar Manfreda, K. (2016). The effect of email invitation elements on response rate in web survey within an online community. Computers in Human Behavior, 56 (C), 320-329. doi10.1016/j.chb.2015.11.025

Ravert, R. D., Gomez-Scott, J., & Donnellan, M. B. (2015). Equivalency of paper versus tablet computer survey data. Educational Researcher, 44(5), 308-310. doi: 10.3102/0013189X15592845

Roberts, L. D., & Allen, P. J. (2015). Exploring ethical issues associated with using online surveys in educational research. Educational Research and Evaluation, 21(2), 95-108. doi: 10.1080/13803611.2015.1024421

Schonlau, M. Fricker, R.D., & Elliot, M. N. (2002). Conducting research surveys via mail and the web. Santa Monica, CA: Rand.

Shannon, D. M., & Bradshaw, C. C. (2002). A comparison of response rate, response time, and costs of mail and electronic surveys. Journal of Experimental Education, 70(2), 179-92.

Sheehan, K. B. (2001). E-mail survey response rates: A review. Journal of Computer Mediated Communication, 6 (2). Retrieved from http://jcmc.indiana.edu/vol6/issue2/sheehan. html

Sheehan, K. B., & McMillan, S. (1999). Response variation in e-mail surveys” An exploration. Journal of Advertising Research, 39(4), 45- 54.

Shih, T. H., & Fan, W. X. (2008). Comparing response rates from web and mail surveys: A meta analysis. Field Methods, 20, 249-271.

Silva, S.C., & Durante, P. (2014). Suggestions for international research using electronic surveys. The Marketing Review, 14 (3), 297- 309.

Spruyt, B., & Van Droogenbroeck, F. (2014). Forewarned is forearmed? A survey- experiment concerning the impact of pre- notification letters on response in a postal

Saleh & Bista

74

survey. Irish Journal of Sociology, 22(2), 86- 95.

Trespalacios, J. H., & Perkins, R. A. (2016). Effects of personalization and invitation email length on web-based survey response rates. TechTrends, 60(4), 330-335.

Trouteaud, A. R. (2004). How you ask counts: A test of Internet-related components of response rates to a web-based survey. Social Science Computer Review, 22, 385-392.

Uhlig, C.E., Seitz, B., Eter, N., Promesberger, J., & Busse, H. (2014). Efficacies of Internet ased digital and paper-based scientific surveys and the estimated costs and time for different sized cohorts. PIOS ONE, 9 (10), 1-11.

Vance, M. (2011). 7 questions consider regarding survey response rates. Research and Marketing Strategies Inc. Retrieved from https://rmsresults.com/2011/01/06/7-

questions-to-consider-regarding-survey- response-rates-marketing-research-in- syracuse-ny/

Van Mol, C. (2017). Improving web survey efficiency: the impact of an extra reminder and reminder content on web survey response. International Journal of Social Research Methodology, 20(4), 317-327. doi:10.1080/13645579.2016.1185255

Veen, F. V., Göritz, A. S., & Sattler, S. (2016). response effects of prenotification, prepaid cash, prepaid vouchers, and postpaid vouchers. Social Science Computer Review, 34(3), 333-346.

Yan, Z., & Fan, W. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behaviors, 26, 132-139.

Copyright of Journal of MultiDisciplinary Evaluation is the property of Western Michigan University, The Evaluation Center and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.