u edwards

2

The Rising Costs and Challenges of Recruiting Research Participants: The Role of Technology and Data Commodification

Student’s name

Institution affiliation

Course

Instructor’s name

Date

The Rising Costs and Challenges of Recruiting Research Participants: The Role of Technology and Data Commodification

Recruiting participants for studies in the past fifty years has grown more expensive and challenging with technological advancements and commercialization of individuals' data. Blumberg and Luke (2022) clarify that "For January-June 2022, 69.8% of adults were wireless-only adults" (1). This shift indicates how rapidly the method of communication has evolved, thereby making the effectiveness of using traditional methods such as telephone interviews less effective. This is all highlighted by the rising ubiquity of mobile phones, call-blocking technologies, and spam filters that further complicate researchers reaching prospective participants.

However, more participant difficulty is not solely due to technology. The rise of data brokers, which collect and exchange personal information, has immensely stoked distrust of research solicitations. The majority of people no longer view unsolicited calls, emails, or street intercepts as indicative of true research but, rather, as spam, a scam, or aggressive marketing. For Tuckel and O'Neill (2022) a higher percentage of survey non-respondents identified distrust of unknown callers as their primary reason for refusing to answer. As a result, potential participants are more remote and harder to engage, and the cost and logistical problems of study administration increase. Now, researchers are having to spend more on recruitment incentives, specific advertising, and ethical disclosure in order to increase response rates.

Amazon’s Mechanical Turk (MTurk) as a Recruitment Solution

One of the creative solutions to such recruitment problems is Amazon's Mechanical Turk (MTurk), a crowdsourcing website launched in 2005. MTurk allows researchers to hire workers (termed "Turkers") to complete Human Intelligence Tasks (HITs), including surveys and experiments (Aguinis et al., 2021). MTurk has been used extensively in academic research due to its affordability and capacity to source large, heterogeneous samples within a few days. According to Aguinis et al. (2021), among the benefits of MTurk is that it accommodates various research designs (e.g., experimental, longitudinal) and allows access to participants with a particular demographic or experience profile.

It is less expensive and quicker than conventional recruitment. For instance, a survey of 1,000 respondents takes days on MTurk whereas months with conventional methods (Follmer et al., 2020). Researchers can also access special populations by filtering by particular requirements, e.g., profession, health status, or political party affiliation (Aguinis et al., 2021). Despite such benefits, MTurk has some validity issues confronting researchers.

Validity Concerns with MTurk Samples

Internal validity threats—elements that undermine causal inferences—are participant inattention, high attrition rates, uneven English skills, and non-naivety (extended exposure to similar studies). As many as 30% of MTurk employees succeed in minimum attention checks, raising issues of data quality, report Aguinis et al. (2021). External validity threats, which relate to the generalizability of findings, are misrepresentation of sociodemographic features and self-selection bias. For instance, when Turkers report their political party, education level, or race inaccurately, the sample will no longer be representative of the target population. In addition, the proliferation of bots and server farms—computer clusters that complete HITs without the bounds of locations—once again compromises data integrity. Farrell and Sweeney (2021) explain that fraudulent accounts are one of the origins of some of the MTurk responses. As countermeasures to these problems, researchers are advised to apply validity checks such as CAPTCHAs, honeypots (hidden response decoys), and attention filters. Cross-validation of worker profiles and monitoring response times can also be applied to identify fraudulent participants.

Demographic Skew and Representativeness

Even though MTurk samples are more likely to be representative than samples of college students, there remain groups that are overrepresented, such as women, White participants, college-educated adults, liberals, and young groups (Lindsay & Vuolo, 2021). This skewing can influence findings, particularly for studies that require diverse opinions. For example, criminal justice reform studies may yield different results if the sample is disproportionately comprised of politically liberal respondents. The increase in internet access mitigated some of the web sample biases, but problems still exist. Criminal justice researchers have yet to explore adequately crowdsourced opt-in sample utilization. These websites are convenient to study public opinion on topics like environmental justice, immigration, or police reform. However, their cons, which are demographic bias and self-selection bias, can bias findings. For example, if police reform studies are drawn from an MTurk sample biased toward young, liberal, college-educated individuals, the results may not reflect attitudes in society at large.

Critical Thinking Applications in Research

Crowdsourced websites are especially well-suited to justice-related research that examines attitudes, behavior, or policy attitudes. Examples are public attitudes toward sentencing law, trust in criminal justice agencies, or perceptions of discrimination (Fine et al., 2025). Issues involving highly technical or vulnerable populations—such as prison inmates, victims of violent crime, or illegal aliens—may not be appropriate for MTurk due to ethical and practical constraints.

For instance, by means of an environmental justice survey administered through MTurk, if a sample is overweighted with affluent, Whites, then findings will tend to fall short in representing problems faced by marginalized communities which are disproportionately subject to pollution (Moss et al., 2023). Similarly, immigration opinion surveys may be skewed if a sample is uneven in political affiliation or cultural identity.

Conclusion

While MTurk offers a cost-effective, streamlined data collection approach, researchers will have to systematically undermine its threats to validity. Meticulous design decisions, strict screening, and disclosure of the sample limit are all part of producing valid, generalizable results. As the era of continuous technological advancement transforms research practices, researchers will struggle with the challenge of maintaining rigor while expanding access to diverse groups. Based on the Scriptures teaching, "Trust in the Lord with all thine heart, and lean not unto thine own understanding. In all thy ways acknowledge him, and he shall direct thy paths." (Proverbs 3:5-6, KJV). Similarly, researchers should be prudent in selecting methodologies, stay ethically grounded, and be cognizant of the limitations of their tools.

References

Aguinis, H., Villamor, I., & Ramani, R. S. (2021). MTurk research: Review and recommendations.  Journal of management,  47(4), 823-837. https://hermanaguinis.com/pdf/JOMMTurk.pdf

Blumberg, S. J., & Luke, J. V. (2022). Wireless Substitution: Early Release of Estimates from the National Health Interview Survey, January-June 2022. https://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless202212.pdf#:~:text=For%20January%2DJune%202022%2C%2069.8%%20of%20adults%20were%20wireless%2Donly%20adults.&text=In%20the%20first%20six%20months%20of%202022%2C,at%20least%20one%20wireless%20telephone%20(Table%201).

Farrell, M., & Sweeney, B. (2021). Amazon’s MTurk: A currently underutilised resource for survey researchers?  Accounting, Finance & Governance Review,  27. https://afgr.scholasticahq.com/article/22019

Fine, A., Berthelot, E. R., & Marsh, S. (2025). Public Perceptions of Judges’ Use of AI Tools in Courtroom Decision-Making: An Examination of Legitimacy, Fairness, Trust, and Procedural Justice.  Behavioral Sciences,  15(4), 476. https://www.mdpi.com/2076-328X/15/4/476

Follmer, D. J., Sperling, R. A., & Suen, H. K. (2020). The role of MTurk in education research: Advantages, issues, and future directions.  Educational Researcher,  46(6), 329-334. https://journals.sagepub.com/doi/abs/10.3102/0013189X17725519

King James Bible. (2017). King James Bible Online. https://www.kingjamesbibleonline.org/ (Original work published 1769)

Lindsay, S. L., & Vuolo, M. (2021). Criminalized or medicalized? Examining the role of race in responses to drug use.  Social Problems,  68(4), 942-963. https://rare.rice.edu/sites/g/files/bxs4146/files/inline-files/Lindsay%20and%20Vuolo%202021.pdf

Moss, A. J., Rosenzweig, C., Robinson, J., Jaffe, S. N., & Litman, L. (2023). Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages.  Behavior Research Methods,  55(8), 4048-4067. https://link.springer.com/content/pdf/10.3758/s13428-022-02005-0.pdf

Tuckel, P., & O’Neill, H. (2022). The vanishing respondent in telephone surveys.  Journal of Advertising Research,  42(5), 26-48. https://www.tandfonline.com/doi/abs/10.2501/jar-42-5-26-48