Human vs Algorithmic Recommendations in the Labor Market: Evidence from a Field
Experiment
Especially in the sourcing and screening of potential candidates, algorithms are
increasingly working alongside human HR staff or replacing them altogether. In view of
findings of “algorithm aversion” or “algorithm appreciation” in other domains, the question
arises whether the recommendations made by an algorithm are perceived differently from those
made by a human (expert), and potentially lead to a different outcome of the hiring process.
We study this question in a preregistered field experiment with law firms. Specifically, we test
whether there is observable difference in employers’ evaluation of candidates recommended
by algorithms relative to those recommended by human advisors. We also elicit preexisting
attitudes and beliefs about certain characteristics of algorithms in a labor market context and a
general preference for algorithmic vs human advice. This allows us to investigate whether and
how potential differences in the evaluation of candidates depending on their label are related
to these individual beliefs and preferences. Results show no overall difference in employer’s
response to resumes recommended by algorithms and humans. In the analysis of heterogeneity
of preexisting attitudes and beliefs towards algorithms, we do not find an interaction of the
treatment effect with the labor market specific beliefs about algorithms. It is rather the general
preference for algorithms which seems to matter for the qualification rating of a candidate.
Decision-makers with a general preference for algorithmic advice also give significantly higher
ratings if a candidate was recommended by an algorithm compared to when the candidate was
recommended by a human resource expert.
Algorithms are becoming ever more important in many domains of life. They are
increasingly being applied to tasks that were previously reserved for and performed by humans
in decision-making processes that range from medical analysis, jail-orrelease decisions,
forecasting employee performance and streamlining the screening of applications to job
openings. While there is considerable literature exploring algorithm efficiency in performing
such tasks, a burgeoning strand of research is now focused on investigating the human aspects
of algorithms, i.e., acceptance or rejection of algorithmic generated insights by individual users
of decision aids. Findings of such interdisciplinary research remain inconclusive, suggesting
that the acceptance of algorithmic advice depends on aspects of the decision environment.
Within this literature strand, one area that has received comparatively little attention is the labor
market. Exploring the factors affecting decision-making in hiring processes has long been of
key concern for policy-makers, given the consequential nature of such decisions for the labor
force structure in the long term. Even minor aspects, such as the sequencing of candidates
evaluated in a hiring process, can have substantial impact in the evaluation of candidates
(Radbruch & Schiprowski, 2020).
The adoption of algorithms in various stages of the hiring process is now widespread.
The technology is widely used for streamlining the screening of applications to job openings
in online environments (Horton et al., 2021) and for recruiting and tracking employees’
performance in influential firms such as Google, Microsoft, and SAP (Walker, 2012). In a
recent industry survey (Spar & Pletenyuk, 2018), 76% of respondents stated that artificial
intelligence will have a significant impact on recruiting. The CEO of ZipRecruiter, a large
platform matching job seekers and employers, estimates that at least three-quarters of all CVs
submitted for job positions in the US are screened by algorithms (Schellmann, 2022). As the
adoption of algorithms in hiring expands, its effects on the labor market also started to be of
concern to policy-makers. Recent regulatory proposals in the United States (US) and the
European Union (EU) call for algorithmic impact assessments of employment decisions. In the
United States, the “Algorithmic Accountability Act of 2022” requires impact assessments of
automated decisions or judgments that have any significant effect on employment (Booker et
al., 2022). In the EU, the Regulation on Artificial Intelligence (AI) also calls for “conformity
assessments” of AI systems used for screening or filtering applications in employment
decisions (European Commission, 2021). Despite the prevalence of algorithms in hiring
decisions and the relevance of its effects in the labor market in the coming years, few are the
studies that have explored it. In view of findings of “algorithm aversion” or “algorithm
appreciation” in otherdomains, the question arises whether the recommendations made by an
algorithm are perceived differently from those made by a human (expert), and potentially lead
to a different outcome of the hiring process.
We study this question in a preregistered field experiment with law firms. Field
experiments, including resume audit studies, have become traditional methods to explore
factors affecting employer’s decisions in the hiring process and have generated robust findings
on how employers respond to resumes characteristics.1 Specifically, we test whether there is
an observable difference in employers’ evaluation of candidates recommended by algorithms
relative to those recommended by human experts. We further explore how employers’ previous
attitudes regarding algorithms and humans might alter how they respond to advice generated
by these sources. The setting of the experiment is a big job fair for undergraduate and graduate
students in Brazil organized by an educational institution. Students send their CVs to this
institution whose HR personnel then decide which students will be matched with which firm
for an interview and forward the respective CV to the respective firm. The firm then interviews
these candidates at the job fair.
Our intervention affects the labelling of the CVs. We randomized whether potential
candidates were presented to the law firms as recommended by a human resource advisor of
the institution or an algorithm. Decision makers at the firms were then invited to participate in
a survey before and after the job fair. In the survey before the job fair, they were asked to rate
the qualification and their interest in hiring each candidate after evaluating their CVs. We also
elicit preexisting attitudes and beliefs about certain characteristics of algorithms in a labor
market context and a general preference for algorithmic vs human advice (i.e., across all
domains of life) in this survey. This allows us to investigate whether and how potential
differences in the evaluation of candidates depending on their label are related to these
individual beliefs and preferences. In a follow-up survey after the job fair, employers were
asked for each candidate whether they had offered a job or an internship to this candidate or
kept the resume for future hiring after the interviews. Results show no overall effect of the CV
label. While characteristics such as work experience or speaking English do affect the rating of
a candidate’s qualification and the hiring interest, the source of recommendation does not. We
also do not find an overall effect in our follow-up survey on job offers or CV retention.
Concerning labor market specific beliefs about algorithms, decision makers on average believe
that human resource specialists are better at taking non-standard profiles in terms of academic
or professional background into account and more trustworthythan algorithms. However,
respondents also believe that human resource specialists are more prone to bias in their
selection of candidates. Concerning recommendations in general across all domains of life, our
respondents prefer human over algorithmic advice.
In the analysis of heterogeneity in the treatment effect along these dimensions, we do not
find an interaction of the treatment effect with the labor market specific beliefs about
algorithms, potentially reflecting the fact that most participants do not believe that one source
of recommendations dominates the other in all three dimensions we elicited. It is rather the
general preference for algorithms which seems to matter for the qualification rating of a
candidate. Decision makers with a general preference for algorithmic advice also give
significantly higher ratings if a candidate was recommended by an algorithm compared to when
the candidate was recommended by a human resource expert. Our findings suggests that firms
introducing algorithms into their hiring process should consider that preexisting attitudes
concerning algorithms can affect the evaluation of candidates depending on the source of
recommendation. The remainder paper is organized as follows. Section 2 discusses the related
literature. Section 3 describes the study’s data and research design. Section 4 presents the main
results, and section 6 discusses our findings and concludes.
This work relates to two different strands of literature. First, we contribute to research
investigating the human side of algorithms, i.e., how individuals perceive algorithmic
recommendations more broadly. Research examining individuals’ perceptions of automated
outputs is not recent.2 Early studies already reported that people reacted to mathematical or
machine problem-solving with skepticism compared to human specialists in medical
predictions and forecasting tasks (Dawes et al., 1989; Meehl, 1954). More recently, this
skepticism to mathematical and computational approaches has been labeled as algorithm
aversion (Dietvorst et al., 2015; Yeomans, 2019). While there is by now a very large literature
exploring the human side of algorithms, findings remain inconclusive. Many studies suggest
that, depending on aspects of the decision environment, individuals can exhibit anything
ranging from extreme aversion to appreciation for algorithms.3 We contribute to this strand of
research in two different ways. First, we explore how individuals perceive algorithm as
opposed to human recommendations in an field experiment in the labor market. Second, we
investigate therole of previously suggested channels influencing algorithm acceptance or
rejection in hiring decisions. Burton et al. (2020) emphasized that individuals’ perceptions and
previous expectations on algorithms and humans might influence how they utilize the
recommendation of such intermediary. Rarely, if ever, human decision-makers will make use
of an algorithm recommendation without bringing their preexisting perceptions and attitudes
regarding what algorithms and humans are capable of doing, given their attributes. What
follows is that decision-makers may respond to a recommendation made by algorithms
differently than to a recommendation made by humans, even if the recommendation is
otherwise identical.
We thus explore how previous attitudes towards algorithms and human advisors might
influence the use of recommendations in hiring decisions. There are several aspects of human
decision-making for which either an algorithm or a human might be more suited. We draw on
the literature to identify relevant dimensions for which firms’ previous attitudes towards
algorithmic and humans could interact with the evaluation of algorithm and human
recommended candidates. A crucial question is if firms perceive humans as more trustworthy
than algorithms at selecting candidates than human resources specialists. Previous studies show
that people tend to distrust automated systems (Underhaug & Tonning, 2019; Muir, 1987; Prahl
& Van Swol, 2017) and often choose human over algorithm advisors (Dietvorst et al., 2015,
2018). Moreover, trust in algorithms is particularly low for subjective or judgmental tasks,
which have no demonstrably best answer (Castelo et al., 2019). A recent survey found that the
majority of Americans find the use of algorithms for CV screening unacceptable (Smith, 2018).
Trust in algorithms or humans is thus a key aspect potentially influencing the use of algorithmic
and human aids to select candidates in hiring procedures.
Another important dimension is the perceived susceptibility of algorithms and humans
to bias. Fairness in algorithmic decision-making is a key concern for individuals, particularly
when evaluating, selecting and hiring personnel.4 Algorithms have created hopes for
overcoming human advisor biases. In the heuristics and biases literature, algorithms are
perceived as a cognitive fix for inherent human limitations of data processing and the errors
and biases related to this (Sundar & Nass, 2001; Kahneman et al., 2021). The ability to process
data and perform accurate and objective predictions in comparison to human analysis is seen
as a motivation for reliance on algorithms. Kleinberg et al. (2020) argue that because of its
greater level of specificity when compared to human decision-making, algorithms could
potentially enhance the detection of biases and the prevention of discrimination in markets such
as labor hiring.
However, while some assert that algorithms may overcome human bias in human
resource decisions, others understand that people may also perceive the decisionmaking as too
simplistic - as if some background or information is not being considered (Newman et al.,
2020). Related to this, algorithm aversion is particularly noticeable in moral domains (Gogoll
& Uhl, 2018). Bigman & Gray (2018) showed that people are more averse to machines on
moral tasks because of the perception that algorithms cannot account for human characteristics.
In particular, it has been suggested that algorithms are ineffective at analyzing outliers
(Germann & Merkle, 2019) and are less able to deal with unique characteristics and
circumstances when compared to humans (Longoni et al., 2019). A straightforward question is
thus whether people perceive algorithms as capable of dealing with exceptional cases in
resumes when compared to humans.
It is also important take into consideration that although many companies are increasingly
using algorithms in the hiring process, there still seems to be particularly low levels of
knowledge regarding the use of algorithms for such task. A study in the United Kingdom,
Germany, France, Poland, Spain and Italy showed that only 31 percent of the population knew
that algorithms are often used to select candidates in hiring processes (Grzymek & Puntschuh,
2019). Such low levels of knowledge prevail regardless of the country. What manifests from
these low levels of awareness regarding the use of algorithms on the labor market is that people
might be influenced by expectations created by the experience with algorithmic aids in domains
that are subject of extensive media coverage and where people are more likely to notice the
consequences in their everyday lives. It is thus important to take a step aside from the labor
market and investigate previous attitudes towards algorithms or humans in areas beyond the
decision domain, as noted by Burton et al. (2020). We have thus also elicited measures of
overall preference for algorithms or human recommendations and investigated how this general
preference can influence the use of algorithmic and human aids to select candidates in hiring
procedures.
Our paper also contributes to a growing literature exploring the introduction of
algorithmic recommendations in the labor market.5 Many studies examine the effect of
introducing automated recommendations on the firm’s decision to interview or hire an
applicant. Barach et al. (2019) analyses the effect of introducing algorithmic recommendations
for improving matching efficiency in various stages of the hiring process, using quasi-
experimental methods. Abebe et al. (2017) goes further and explores if job fairs improve
employment outcomes using a randomized control trial. They randomize invites to both
workers and firms to participate in the job fair, and then an algorithm is used to match them.
Horton (2017) conducts an experiment in a online platform and finds that recommendations
increase hiring. In a small field experiment, Cowgill (2018) shows that by replacing the
decision of a human CV screener by an algorithm, selected candidates were more likely to pass
interviews and receive job offers. All of these studies, however, overlook the role of traditional
human recommendations in the labor market. They overlook, in particular, potential differences
in the effect of algorithmic and human recommendations. In other words, they all analyze the
effect of introducing algorithmic recommendations in the hiring process, by using as a
counterfactual no recommendation at all. To address this gap, our work pursues a
complementary aspect to this literature, using a randomized control trial to analyse if and how
human vs. algorithmic recommendations could lead to different employment outcomes. To the
best of our knowledge we are the first to conduct a field experiment to study human vs.
algorithmic recommendation in the labor market.
