Value Sensitive Design in policy making or IT

SPECIAL ISSUE: ICT AND SOCIETAL CHALLENGES

THE DUALITY OF EMPOWERMENT AND MARGINALIZATION IN MICROTASK CROWDSOURCING: GIVING VOICE TO THE LESS

POWERFUL THROUGH VALUE SENSITIVE DESIGN1

Xuefei (Nancy) Deng College of Business Administration and Public Policy, California State University, Dominguez Hills,

1000 E. Victoria Street, Carson, CA 90747 U.S.A. {ndeng@csudh.edu}

K. D. Joshi Carson College of Business, Washington State University, Todd Hall 442,

Pullman, WA 99164-4743 U.S.A. {joshi@wsu.edu}

Robert D. Galliers Information and Process Management/Sociology Departments, Bentley University, 174 Forest Street, Waltham, MA 02452 U.S.A. and

School of Business and Economics, Loughborough University, Loughborough, Leicestershire LE11 3TU U.K. {rgalliers@bentley.edu}

Crowdsourcing (CS) of micro tasks is a relatively new, open source work form enabled by information and communication technologies. While anecdotal evidence of its benefits abounds, our understanding of the phenomenon’s societal consequences remains limited. Drawing on value sensitive design (VSD), we explore microtask CS as perceived by crowd workers, revealing their values as a means of informing the design of CS platforms. Analyzing detailed narratives of 210 crowd workers participating in Amazon’s Mechanical Turk (MTurk), we uncover a set of nine values they share: access, autonomy, fairness, transparency, communi- cation, security, accountability, making an impact, and dignity. We find that these values are implicated in four crowdsourcing structures: compensation, governance, technology, and microtask. Two contrasting perceptions—empowerment and marginalization—coexist, forming a duality of microtask CS. The study con- tributes to the CS and VSD literatures, heightens awareness of worker marginalization in microtask CS, and offers guidelines for improving CS practice. Specifically, we offer recommendations regarding the ethical use of crowd workers (including for academic research), and call for improving MTurk platform design for greater worker empowerment.

1

Keywords: Crowdsourcing, societal impacts, crowd worker value, ICT ethics, empowerment, marginalization, value sensitive design, open source, Amazon’s Mechanical Turk, microsourcing, gig economy, on-demand workforce

“There’s no place for us to be heard if we’re taken advantage of or treated unfairly. We’re many, and invisible, and easily replaced. So we’re ignored.”

(A 33 year-old MTurk worker)

1Ann Majchrzak, M. Lynne Markus, and Jonathan Wareham were the accepting senior editors for this paper.

The appendix for this paper is located in the “Online Supplements” section of the MIS Quarterly’s website (http://www.misq.org).

MIS Quarterly Vol. 40 No. 2, pp. 279-302/June 2016 279

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Introduction

Advancements in information and communication tech- nologies (ICT) are impacting many facets of our society. One such impact is apparent in the rapidly growing phenomenon of crowdsourcing (CS), an open source work form, enabled and mediated by the Internet and social media. CS is the practice of obtaining needed services and content by soliciting voluntary contributions in the form of an open call from a large network of individuals rather than from an organiza- tion’s employees or suppliers (Howe 2006). CS occurs in such forms as micro work, creative CS, and inducement prize contests (Howe 2008). It radically changes the nature of work: rather than being confined to offices and stipulated office hours, people can conduct work at home, choose when to work, and decide which jobs to perform. CS, thus, appears very attractive—on the surface, that is.

CS has attracted wide attention in industry and academia. Anecdotal evidence indicates that CS is changing people’s perspectives on managing their work–life balance. Prominent articles in the popular media include one in The New York Times on how individuals are able to participate in social change through CS (Rosenberg 2011), and an announcement by Ladies’ Home Journal to crowdsource its publishing content to readers rather than relying solely on professional journalists (Sivek 2012). The demand for microtask CS is one of the most rapidly expanding trends (Bratvold 2011). Compared to regular jobs in a “traditional” organization, these micro tasks are simple (e.g., can be completed in a matter of minutes) and are compensated with tiny monetary rewards. According to a survey of registered workers on Amazon’s Mechanical Turk (MTurk), the average U.S. MTurk worker earned $2.30 per hour in 2009 (Ross et al. 2010). Academic work includes Deng and Joshi (2013), who explored crowd workers’ perceptions of microtask CS as a career choice, finding that work flexibility and work autonomy were the two major positive aspects. They also highlighted concerns about workforce marginalization, however, given commonplace complaints regarding low levels of compensation.

It therefore remains unclear whether microtask CS provides a platform that empowers workers to craft their careers or creates a sweatshop environment where workers are com- pleting fragmented tasks for minimal pay. Thus, although CS can afford worker autonomy and flexibility, it can also make workers vulnerable to exploitation. The power asymmetry provides opportunities for abuse, with fairly benign work- shops potentially degenerating into digital sweatshops: “given the short time commitment between crowd worker and requester, it is easy to imagine heightened exploitation and dehumanization” (Kittur et al. 2013, p. 10).

In line with Desouza, Ein-Dor et al. (2007), Desouza, El Sawy et al. (2006), and Wastell and White (2010), the onus of criti- cally evaluating the microtask CS phenomenon to direct its use to empower crowd workers and prevent its misuse lies, in part at least, on us as IS scholars. This special issue on societal challenges of ICT offers an opportunity for just such critical reflection. Hence, we give voice to the crowd workers who are less powerful but most affected by the values implicated in the design of microtask CS platforms. The primary goal is to reveal causes of the rising concerns related to microtask CS and to propose research that focuses on means by which these may be ameliorated. Broadly, the objective is to highlight evidence of empowerment and marginalization in CS environments, which could then lead to uncovering means of accentuating worker empowerment while reducing worker marginalization. Thus, our study responds to the challenges posed by this special issue by (1) providing a rich description of an emerging paradox of worker empowerment and marginalization in this context, (2) advancing theoretical understanding of the societal chal- lenges of this emerging phenomenon, and (3) proposing a novel, ethical design perspective for incorporating moral import to cope with associated societal challenges.

To explore worker values that could be foundational in designing CS platforms, we draw on value sensitive design (VSD) (Friedman 1996; Friedman and Khan 2003; Friedman et al. 2006) in analyzing the narratives of MTurk crowd workers. As a well-established online CS marketplace with a large pool of job seekers and requesters, and a variety of work, MTurk makes for a suitable context for our study. Our analysis reveals a set of nine worker values (access, autonomy, fairness, transparency, communication, security, accountability, making an impact, and dignity) implicated in the four CS structures (of compensation, governance, tech- nology, and microtask), which are associated with perceptions of empowerment and marginalization. We explain why these dual impacts of CS may be inseparable and propose future research directions based on our findings. We do so with the objective of reducing power asymmetries among crowd workers, job requesters, and those who host CS platforms. This, we argue, could result in fostering a more trusting work environment where crowd workers are better appreciated and less resentful. Theoretically, we advance the broad field of design research by extending the VSD literature and by intro- ducing a value-centric design perspective. The study also advances theoretical understanding of microtask CS by articulating crowd worker perspectives and values and by revealing the dynamic interrelationships between values and the two opposing experiences of empowerment and marginalization.

The remainder of the paper is organized as follows. Next, we summarize relevant microtask CS literature and review key

280 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

concepts in VSD. We then describe pertinent research methods before presenting our findings. There follows a dis- cussion on the duality of empowerment and marginalization resulting from the interplay between worker values and current CS designs. We then discuss our theoretical contri- butions and outline design and practice implications before concluding the paper by means of a number of reflections.

Microtask Crowdsourcing

Microtask CS is an open source form of micro work for micropayment. Consistent with Howe (2008), we define microtask CS as a type of online, participative activity in which undefined, generally large groups of individuals take on micro tasks posted on a web-based, third-party platform in an open call by organizations or individuals in exchange for micropayment. In addition to MTurk, examples of microtask CS include MobileWorks2 and CrowdFlower.3 These three largest microtask CS platforms aggregate hundreds or thou- sands of tasks performed by multiple suppliers from a large pool of approximately 400,000 workers (Kaganer et al. 2013). We refer to those individuals who perform micro tasks for micropayment as crowd workers, those who post micro tasks as job requesters, and those web-based, third-party platforms as crowdsourcing platforms (or CS platforms).

Based on our CS literature review, research on microtask crowdsourcing can be organized around its four key struc- tures: technology, governance, microtask, and compensation. First, the technology structure refers to the IT infrastructure used to build the CS work environment. Research on tech- nology structure focuses on technical functionalities to meet stakeholder needs. For instance, Kajino et al. (2014) pro- posed a CS quality control protocol to be imbedded in the technical system so as to allow a job requester to assess the quality of results while preserving worker privacy; Saito et al. (2014) proposed a framework of three core modules—tutorial producer, task dispatcher, and feedback visualizer—supported by a back-end skill assessment engine to enable micro-tasking of skill-intensive work; and Geiger and Schader (2014) developed personalized task recommendation mechanisms to better match CS tasks and workers’ individual interests and capabilities. Second, the governance structure refers to CS work practices, standards, and policies. Research has high- lighted the importance of the governance structure in terms of challenges in managing large, external groups of people— referred to as an on-demand, scalable workforce (Greengard

2011) or the “human cloud” (e.g., Kaganer et al. 2013)— handling a variety of tasks (Schenk and Guittard 2011) that have traditionally been associated with small, specialist groups in organizations (e.g., Kittur et al. 2013). Third, the microtask structure refers to the properties of CS jobs. Designs of the CS microtask structure (e.g., instructions, configurations) drive worker participation (Chandler et al. 2013) and enhance worker productivity (Finnerty et al. 2013; Moussawi and Koufaris 2013). Finally, the compensation structure refers to payment arrangements for CS jobs. Scholars (e.g., Finnerty et al. 2013; Kittur et al. 2013) have evaluated the CS compensation structure (e.g., payment rates and monetary rewards for completed microtasks) as an important factor for job requesters, especially when job requesters consider these sourcing opportunities. (For a more detailed review of the CS literature, refer to Appendix A).

While prior studies focus on efficiencies in these four struc- tures, systematic investigations regarding the intended or unintended ethical consequences of these new work structures have been largely absent. Societal concerns regarding ethical standards and practices that primarily focus on microtask compensation structures are increasing (Schmidt 2013; Silberman et al. 2010); job requesters may be using these new work structure forms to bypass commonly established ethical standards to their advantage (Harris 2011), complicating the application of existing work laws to crowd labor (Felstiner 2011). In addition, prior studies, with the exception of Brabham (2012) and Silberman et al. (2010), consider job requester or CS platform owner perspectives, paying little attention to the views of crowd workers. Yet, crowd workers, the key stakeholder, cannot be ignored if one cares about microtask CS’s long-term impact on society. Research at the intersection of ethics and CS work structures is warranted. We aim to fill this gap by improving our understanding of this emerging, complex phenomenon and giving voice to the crowd workers themselves.

Design Theory: Value Sensitive Design of ICT

Microtask CS is a complex online labor marketplace phenomenon that involves multiple stakeholders, thus requiring consideration of multiple user perspectives in CS platform design. Markus et al. (2002) offer a key design principle of identifying critical players and harmonizing diverse perspectives in developing IS for emergent knowledge processes with unpredictable requirements. Given our focus on ethical considerations, and taking the perspective of less powerful workers, we introduce VSD as a means of examining the power asymmetries introduced above.

2https://www.mobileworks.com/

3https://www.crowdflower.com/

MIS Quarterly Vol. 40 No. 2/June 2016 281

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

VSD is a value-oriented design methodology commonly adopted in human–computer interaction (HCI). Developed by Friedman (1996) and Friedman and Khan (2003), VSD is a “theoretically grounded approach to the design of technology that accounts for human values in a principled and compre- hensive manner” (Friedman et al. 2006, p. 2). Studies on ICT and ethics have historically placed emphasis on enduring human values (e.g., Wiener 1950, 1954, 1964). VSD seeks to understand how human values (e.g., welfare, accountability, autonomy, freedom from bias) can be accounted for in the design of computer technologies. Values are not facts, but are derived subjectively, based on the interests and desires of human beings within a sociocultural milieu (Friedman et al. 2008).

In capturing and prioritizing human values in design, VSD proposes a three-part methodology that includes conceptual, empirical, and technical investigations to guide design (Friedman and Khan 2003; Friedman et al. 2006). These investigations are applied iteratively, with findings from new investigations building on earlier results. Conceptual investi- gations focus on theoretically and philosophically informed analyses of central constructs and issues, which result in working conceptualizations of the values under investigation. The empirical work seeks to understand human responses to technical artifacts and to the larger social context of tech- nology use. Technical investigations can be either retrospec- tive analyses of existing technologies or proactive design of systems to support values identified in the conceptual or empirical investigations. The distinction between the tech- nical and empirical investigations lies in their unit of analysis. Technical investigations examine the technology; empirical investigations capture the responses of individuals, groups, or communities that are involved in and/or affected by the technology (Friedman et al. 2008).

We use the human value constructs of VSD to theoretically and philosophically underpin our empirical investigation. The technical investigation (undertaken with a view to support crowd worker values through retrospective or prospective assessments) of the MTurk platform (comprising multiple work structures) can be undertaken only after salient values are identified in the conceptual and/or empirical investigation. VSD research commonly relies on case studies in discussing how to account for human values in ICT designs. Table 1 summarizes illustrative VSD studies.

Friedman and Kahn (2003) provide a classification of values (referred to as a collection of 12 “human values with ethical import”): human welfare; ownership and property; privacy; freedom from bias; universal usability; trust; autonomy; informed consent; accountability; identity; calmness; and environmental sustainability. This set of values, as Friedman et al. (2006) indicate, is open to refinement. While appre-

ciating the positive impact of VSD on technology design, Le Dantec et al. (2009) point to its limitations, viewing VSD as a design methodology,

promulgating an agenda of design on a largely fixed classification of values, rather than inquiring about the values present in a given context and responding to those values—being sensitive to those values— through design (p. 1143).

Thus, they argue that user values should be empirically revealed before being used to design or refine systems. Consistent with this argument, we conduct this exploratory study to reveal crowd worker values that can be embodied in the design of CS work structures (e.g., governance, tech- nology, compensation, and task). We now turn to these research considerations.

Research Methods

We conducted an in-depth, interpretive field study to inves- tigate which aspects of CS platforms are appreciated by crowd workers and which contribute to empowerment and marginalization in microtask CS. Consistent with interpretive approaches to IS research (as outlined, for example, by Galliers and Land 1987; Orlikowski and Baroudi 1991; Walsham 1995), our research objectives were to investigate how human actors (crowd workers) made sense of their participation in the CS work environment, rather than to hypothesize or test cause-and-effect relationships. Informed by earlier theorizing and by our empirical study, our goal was to develop an analytical generalization regarding worker values and their experiences with the open, online labor marketplace mediated and enabled by ICT. This generali- zation could prove useful for further research on other types of CS and ICT-enabled work contexts. Our approach is consistent with Klein and Myers’ (1999) principle of abstrac- tion and generalization for interpretive field studies, and Lee and Baskerville’s (2003) framework for generalizability (i.e., empirical to theoretical generalization).

Amazon’s Mechanical Turk (MTurk)4 provided an oppor- tunity to collect rich case study data in a setting where the

4The MTurk platform has been increasingly used by researchers from dif- ferent disciplines for field experiments. Examples include Alonso and Mizzaro (2012) on using the MTurk platform as a cheap, quick, and reliable alternative for relevance assessment of information retrieval, and Chandler and Kapelner (2013) for running natural field experiments in economics. Mason and Suri (2012) provide detailed guidelines on how to use Amazon’s MTurk website to conduct behavioral research. Utilizing such platforms as MTurk as an alternative to student surveys is proposed in Steelman et al. (2014).

282 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table 1. Summary of Reviewed Studies on Value Sensitive Design

Reference Technology Value Objectives and Outcomes

Friedman et al. 2002

Mozilla browser Informed consent in online interactions Development of new technical mechanisms for cookie management in a web browser.

Friedman et al. 2006

(1) Web browser; (2) high- definition; plasma display; (3) urban planning simulation system

(1) Information and control of web browser; (2) physical and psychological well-being and privacy in public spaces; (3) diverse range of values by different stakeholders (environmental sustain- ability, business expansion opportunity, neighborhood walking safety).

Engaging VSD in the design of computer systems for various stakeholders.

Miller et al. 2007

Groupware system Privacy, awareness, and reputation Technology and organizational policy coevolve. New design methods (value dams and flows).

Le Dantec et al. 2009

(1) Mobile technology use by homeless; (2) RFID in passports, credit cards and retailers; (3) home technology

(1) Staying connected with family and friends; independence (or autonomy); (2) justice and accountability; (3) quality, durability, and sustainability.

Attending to and engaging local expressions of values (the values present in the technology use context) in the design.

phenomena we hoped to observe were likely to be widespread (Yin 1994), and allowed us to observe first-hand the different aspects of worker empowerment and marginalization from the perspectives of crowd workers themselves. Our data analysis was based on such qualitative research methods (open coding, analytical categories informed by prior research, data display matrices) as articulated by Miles and Huberman (1994). The following section describes the research site before detailing data collection, coding, and analysis.

Research Site and Microtask CS Description

MTurk is a platform that offers access to large numbers of job requesters and crowd workers to engage in a variety of micro tasks. MTurk provides free access and services to crowd workers by allowing them to select and perform tasks for monetary reward. According to a recent browsing of the MTurk website, there are now more than 400,000 workers registered and over 395,700 tasks available (March 27, 2015) (see also Satzger et al. 2013). The website refers to the micro tasks as “human intelligence tasks” (HITs), which typically involve fairly simple tasks, such as video and audio transcrip- tion, classification, and document categorization.

The MTurk website allows workers to search for HITs using a predefined set of criteria and allows job requesters to create and publish micro tasks by using a web service-based inter- face. The CS platform provides job requesters with the possibility of publishing simple task descriptions in a database to which all workers have access. Task descriptions comprise

a title, HIT type, text description, expiration date, time allotted, keywords, required qualifications, and monetary reward. MTurk recommends (but does not enforce) job requesters to offer $0.1 for one minute’s work, equivalent to $6.00 per hour. At the time of the study, MTurk charges a 10 percent commission5 based on payments made, but charges 30 percent when a micro task requires completion by MTurk Masters. The FAQs posted on the website (https://requester. mturk.com/help/faq) provide general guidelines, including information on the types of HITs that violate MTurk policies (e.g., HITs requiring disclosure of a worker's identity or e-mail address, or asking workers to solicit third parties).

Data Collection

We collected data via MTurk by using a survey instrument (Appendix B), which included both semi-structured and unstructured questions on worker experiences. Rather than directly asking what crowd workers value most in microtask CS work, we employed the indirect approach suggested by Friedman et al. (2006); that is, we asked them to describe how and why they started to participate in the CS workforce and how they felt about taking jobs on MTurk. This approach helps to “engage people’s reasoning about the topic under in-

5Starting July 22, 2015, Amazon increased its charge to Job Requesters from 10% of the reward and bonus amount (if any) that Job Requesters pay workers per HIT to 20% and updated the charge for using each Master Quali- fications to 25% (http://mechanicalturk.typepad.com/blog/2015/06/following- up-on-our-commission-structure.html).

MIS Quarterly Vol. 40 No. 2/June 2016 283

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table 2. Descriptive Statistics and Correlation (n = 210)

Variable Mean (SD) (1) (2) (3) (4) (5)

(1) Age 35.0 (12.2)

(2) MTurk Tenure (Months) 15.2 (16.9) .271***

(3) Weekly HITs 958 (1539) -.047 .084

(4) Weekly Hours 26.1 (16.1) .093 .015 .409***

(5) Gender 0.52 (0.5) .169* .170* -.019 .007

(6) Consider CS as a career (Y/N) 0.47 (0.47) .101 .141* .252*** .389*** .114

vestigation” (ibid., p. 19). In addition, we asked them to share their experiences in completing their favorite types of HIT. The survey also included questions about their MTurk tenure (how long they had been working in CS), CS efforts (hours and HITs on a weekly basis), and their demographics (e.g., gender, age, education, employment status, household income).

The survey was published on the MTurk website in the form of a survey HIT. Workers were able to choose to take the sur- vey and be compensated, as with any other type of HIT. Workers were compensated $2.00 for their response and they spent approximately 16 minutes on average on completing the survey, corresponding to a $7.50/per hour rate.6 The survey was well received, with the following responses being typical:

This is a very interesting survey; it makes me think a lot about why I do the work.

Thanks for the opportunity to think about my place at MTurk!

This is a very interesting survey. I hope you are passing the results of the survey along to [MTurk].

Our data sample included 210 individual responses by U.S.- based MTurk workers (110 females; 100 males) representing a diversity of employment types (full-time employed, part- time employed, otherwise unemployed) and others (student, retired, stay-at-home mom). More than one third of the respondents were employed full-time, and a quarter were employed part-time. On average, they spent 26 hours each week on MTurk, with a median 20 hours for full-time employed and a median 30 hours for unemployed. Table 2 summarizes the sample’s descriptive statistics.

The majority (over 75%) had received some college or higher education—bachelor’s degree (34.8%); some college but no degree (29.5%); graduate degree (11%)—with the remainder holding a high school diploma (12.9%) or an associate degree (11.9%). Annual household income for 39 percent of respondents was $25,000–49,999, with 22.9 percent receiving $50,000–74,999. The distribution of the remaining categories was: < $25,000 (18.6%); $75,000–99,999 (11%), and $100,000+ (8.6%). The average age of respondents was 35 years (SD 12.2) with, on average, 15 months’ (SD 16.9) experience of MTurk. Our survey respondents are distributed in the following age groups: 18–24 years (21%); 25–30 years (24%); 31–40 years (22%); 41–50 years (21%), and 51+ years (11%). The sample demographics7 are consistent with sample demographics of MTurk workers in prior studies (e.g., Berinsky et al. 2012; Goodman et al. 2013; Ross et al. 2010).

MTurk classifies HITs into seven categories: data processing, categorization, sentiment, tagging, content, business feedback, and academic survey. Our respondents undertook multiple types of HIT, with 60.7 percent of them performing all seven types of HIT, followed by 20 percent performing six types of HIT, and 10.7 percent performing five types of HIT. Only 8.7 percent performed four or fewer types of HITs. None under- took only academic or categorization HITs. Table 3 provides definitions and examples of the seven HIT categories.

Data Coding and Analysis

In the initial data coding, we used crowd worker statements to identify values that were expressed in relation to their CS experiences. We adopted Miles and Huberman’s (1994)

6While previous MTurk studies (Horton et al. 2011; Kapelner and Chandler 2010) have shown that monetary remuneration is not a primary motivator for participation in MTurk studies, we set our rate to exceed the current Federal minimum wage.

7Our survey specifically required MTurk workers that are U.S. based. The workers’ mean age of 35 years in our sample is similar to the average age of workers in prior studies on MTurk workers, e.g., 33 years in Goodman et al. (2013) and 32.3 years in Berinsky et al. (2012). The distribution among age groups is similar to that in Ross et al.’s (2010) study, except that our sample included more workers in the 41–50 years (21% versus 11%) and fewer workers in the 18–24 years (21% versus 40%) categories.

284 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table 3. Percentage of Crowd Workers (n = 210) Performing the Seven Categories of MTurk HITs

Percentage* of Respondents HIT Category Definition and Example

98.2% Academic survey Involves completing surveys and participating in scientific studies.

74.5% Categorization Categorizing products or to check data accuracy in catalogs,

63.6% Business feedback

Providing feedback to businesses, such as providing feedback on a company’s website design or new products,

61.8% Sentiment Requires rating sentiments present in tweets, press coverage and customer comments.

58.2% Content Ranging from reviewing and editing content to writing abstracts/articles on specific subjects.

56.2% Data processing Refers to those micro tasks on verifying data entry, collecting data or cleaning duplicate/incorrect data files.

36.2% Tagging Including generating key words for images, advertisements or websites for indexing and searching purposes.

*Note: The percentage indicates the proportion of respondents who performed that category of HITs; a respondent could select multiple categories.

coding strategy, undertaking data coding in multiple steps. First, two researchers determined the coding scheme of value categories based on prior VSD studies (e.g., Friedman and Kahn 2003; Le Dantec et al. 2009) and performed a pilot coding on eight sample responses (two responses from each of the four employment categories). New value categories emerged or were modified as a result. The two researchers then discussed the pilot coding results and refined the coding scheme. They then independently coded a random sample of 60 responses, discussed the coding, and resolved any coding disagreements. Table B1 in Appendix B provides examples of coding discrepancies and their resolution. The inter-rater reliability of coding is satisfactory, with a Cohen’s Kappa Index of 0.885, suggesting an acceptable level of agreement between the two coders (Ryan and Bernard 2000). Together, the two researchers coded one third of the sample, compared and discussed coding, and refined and finalized the coding scheme. Then, one coder followed the agreed coding scheme to complete coding of the remaining data. We assigned be- tween one and eight values to a worker’s statements (average 3.7 values, median 3 values per worker). This iterative pro- cess resulted in nine value categories, summarized in Table 5 in the “Findings” section. Table B2 in Appendix B details the distribution of worker values by employment status, gender, education level, household income, and age group.

The coding of workers’ value statements revealed expressions of empowerment and marginalization. We coded the expres- sion of empowerment into four dimensions (meaning, compe- tence, self-determination, and impact) that are similar to those adopted by prior studies (Spreitzer 1995; Thomas and Velt- house 1990), reflecting an individual’s orientation to his or

her work role.8 For coding instances of marginalization, we started with the definition that, to marginalize means “to put or keep (someone) in a powerless or unimportant position within a society or group” (http://www.merriam-webster. com/dictionary/marginalize). The coded responses that cap- tured this sense of marginalization revealed feeling exploited (economic marginalization), deskilled (competence marginali- zation), constrained by the MTurk technical system (institutional marginalization due to technological features), and helpless vis-à-vis job requesters and the MTurk platform owners (marginalization due to institutional policy and practice). Each worker statement was assigned between zero and four empowerment types (average 2.1; median 2) and between zero and four marginalization types (average 1.3; median 1). The linkage between a coded value and the resulting experience of either marginalization or empower- ment was also captured. The frequencies of marginalization and empowerment instances are summarized in Tables 6 and 7 in the “Findings” section.

Data analysis was undertaken iteratively. We first read the worker narratives and coded the statements to reflect the categories of values and the presence of stated feelings of empowerment or marginalization. For example, our coding revealed workers valuing the flexibility in choosing CS work, suggesting an “autonomy” value enabled by CS task structure (see Table 4). Our reading of the remaining statements revealed workers’ appreciation of flexibility and feeling inde-

8Spreitzer (1995) developed and empirically validated the measurement for the four cognitions that we adopted in coding crowd workers’ feelings of empowerment.

MIS Quarterly Vol. 40 No. 2/June 2016 285

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table 4. A Sample Matrix Illustrating the Data Coding Process

Value Statements: What a crowd of worker considers important in life

Value Revealed

Value Category

Value Implicated in Current Design

Empowerment/ Marginalization

“There are at any time over 200,000 HITs of many different types available .... Since [they] take a short time, I can accept them 24 hours a day at my leisure, I can complete work on my computer anywhere …”

Flexibility to choose a micro task.

Autonomy: Freedom and independence in job decision making.

The variety and micro nature of CS task (task structure) provide flexibility for workers to accept and complete a job in a short time.

Empowered through self-determination (Freedom and independence in choosing and com- pleting CS jobs).

“The pay can be really atrocious, akin to sweatshop wages. It would be nice if everything had a fair wage.”

Value the fair rate of com- pensation for a micro task.

Fairness: Fairness in the rate of payment.

CS micro payment (com- pensation) fails to support fair compensation rate for some micro tasks.

Economic marginalization (feeling exploited).

pendent in accepting and completing HITs, suggesting that undertaking CS jobs is empowering via self-determination. In like manner, we coded workers’ narratives and responses to all questions, highlighting text segments (e.g., phrases, a sentence, or sentences where there were illustrative examples). Then, we revisited the concepts that had been coded and assessed whether they could be combined with other concepts in a category, such as categorizing “self- determination” as “feeling empowered.” Based on the four structures of microtask crowdsourcing discussed in the CS literature review, we also related each value statement to one of the four CS structures: micropayment (compensation), MTurk policies and procedures (governance), CS technical system (technology), and task characteristics (microtask).

We used data display matrices (Miles and Huberman 1994) to record identified concepts and categories and to show patterns (themes) between major constructs (e.g., worker value, psychological experience, CS structure). Table 4 contains a sample data matrix to illustrate the analytical method adopted.

Building on this analysis, we examined how the values expressed were related to experiences of feeling empowered and/or marginalized. We then combined the value-experience relations across the 210 respondents, revealing four over- arching themes that characterized the complex phenomenon of crowd worker engagement in microtask CS: (1) crowd worker value is a multifaceted construct of nine human values implicated in the CS platform’s work structures; (2) empower- ment is manifested in the form of four cognitions when the desired values are adequately implicated on the CS platform; (3) marginalization emerges in four different forms when the desired values are not sufficiently implicated on the CS platform; (4) the two coexisting but contrasting feelings (empowerment and marginalization) are experienced simul- taneously by the same crowd workers, which we characterize as a duality. These overarching themes are discussed in the two sections that follow.

Findings

Crowd Worker Values

The crowd workers surveyed shared a set of nine key values associated with their work-related expectations as they inter- acted with, and engaged in, CS work on MTurk: access, autonomy, fairness, transparency, communication, security, accountability, making an impact, and dignity (Table 5). How these values were implicated in various work structures within the CS platform varied greatly, however.

Access to CS work is overwhelmingly valued but is multi- faceted; it conveys different meanings to workers. It provides a means of income generation for those who are unable to conform to traditional workplace expectations due to certain life circumstances (e.g., stay-at-home parents, individuals with health problems). For those unemployed, CS can be their only job option for making ends meet (i.e., paying bills, buying groceries), creating an important financial cushion. Autonomy emerged as one of the most salient values held by MTurk crowd workers, with them appreciating flexibility and freedom in deciding what tasks to take on, and how, where, and when to perform them. While some enjoyed control over their work schedule, others appreciated the freedom in their choice of tasks. In addition, crowd workers expressed the desire of making an impact, contributing to the community and having a positive impact on others’ lives, such as in performing research-related, survey HITs.

The perception of fairness (or lack thereof) is associated with two important aspects—compensation and governance—of CS work. While most appreciated that they could obtain monetary reward (“The best thing about doing [CS] jobs on [MTurk] is getting paid”), the same individuals felt they were unfairly compensated (i.e., being paid 20 cents for 10 minutes’ work) and sometimes unfairly treated (i.e., job requesters rejecting work without reason). Moreover, some

286 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table 5. Value Categories and Examples

Crowd Worker Value Example Percentage*

Access: Open and equal access to work opportunities offered in the CS environment (derived from the study).

[CS] work is providing me money to pay monthly bills and helping me to dig out of the hole I was in from being out of work for so long. [MTurk] helps me to pay bills and buy groceries. It’s necessary until I get another job that pays more. (Female, 45 years; Bachelor’s degree; Household income $25,000–$49,999; Unemployed)

97%

Autonomy: Ability to decide, plan, and act in ways that are believed will help in achieving personal goals (Friedman 1996). Having a strong sense of freedom and independence in work choices (Schein 1985).

I enjoy being able to choose what I’ll work on. I’ve chosen assignments based purely on pay … and … because the subject matter was interesting to me or for a good cause. (Female, 45 years; Bachelor’s degree; Household income $25,000--$49,999; Employed full-time)

85%

Fairness: The CS work process are unbiased (Modified from Walldius et al. 2005). Freedom from bias; not privileging one person, group, stakeholder over another (Friedman et al. 2006).

It’s important that worker get paid the amount they deserve. I think all requesters should put HITs at a minimum of $6/hr or more. For example, a survey that takes 10 minutes but only pays 20 cents is ridiculous! (Male, 18 years; Some college education; Household income $75,000–$99,999; Unemployed)

60%

Transparency: The process by which CS work standards and protocols are certified to be open and understandable (modified from Walldius et al. 2005).

I’d change how requesters are reviewed and rated so that Turkers like myself can avoid bad requesters and do quality work for the ones that are worth it. For example, as of now, we can’t see ratings of [requesters] on the site… so we’re 'blind' and can’t know that a requester can potentially reject our content. Having a rating platform (like Turkopticon) benefits everyone working for the site. (Male, 25 years; Bachelor’s degree, Household income $25,000–$49,999; Employed full-time)

30%

Communication: The capability to inform others and being informed during CS job processes (derived from the study).

I’d like to see better communication between requesters and workers. Amazon doesn't get involved in disputes or misunderstandings between workers and requesters. For the most part, the requesters who I have needed to contact have been polite and have made a point to respond to me, which I appreciate. (Female, 28 years; Some college education; Household income $50,000–$74,999; Unemployed)

26%

Security: Protecting people’s rights to perform jobs; lack of job security is evidenced by disruption and threat to one’s work environment (modified from Schein 1985).

I’d like MTurk to be more protective of workers—we get scammed a lot. Say you spend 30 minutes filling out a survey, but after you submit your answers, you get no completion code to get paid. They have their data and you get nothing. (Female, 43 years; High school graduate; Household income $75,000–$99,999; Employment-Other: Stay-at-home mom)

20%

Accountability: The properties that ensure that the actions of a person, people or institution may be traced uniquely to the person, people or institution (Friedman and Kahn 1992).

Requesters should be held accountable for their shortcomings/unethical behavior because too often they abuse a system that does not care. (Male, 24 years; Bachelor’s degree; Household income $75,000–$99,999; Employed part-time)

20%

Making an impact: Work influences other individuals, groups, and communities (Schein 1985).

[Working on academic surveys] gives me a sense of pride knowing that I’m helping the research community by assisting them with data collection … I really enjoy surveys … for Master's or Doctoral research, because I know that someone working to not only provide new, insightful research … but to also better themselves. (Male, 30 years; Bachelor’s degree; Household income $75,000–$99,999; Employed part-time)

16%

Dignity: A sense of pride in oneself and self-respect (modified from Le Dantec and Edwards 2008).

I don't feel like there’s enough respect for workers. For example, someone might offer $0.50 for an hour’s … work. Requesters can often be ignorant of the ins and outs of MTurk and this can lead to unwarranted rejects. (Male, 34 years; Bachelor’s degree; Household income $50,000–$74,999; Employed full-time)

11%

*Note: Each worker’s statements may have more than one coded value category assigned so the percentages do not sum to 100%.

MIS Quarterly Vol. 40 No. 2/June 2016 287

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

crowd workers also perceived unfairness in MTurk’s worker evaluations (i.e., MTurk’s Master Qualification). Trans- parency is deeply rooted in the openness of CS work standards and protocols. To some extent, the value of trans- parency is embedded in open source work because each published HIT includes a brief job description, instructions, time requirement, and payment amount. However, some- times, workers found their interactions with job requesters to be less forthcoming, leaving them feeling “blind” to the work process. They expressed their desire for direct and open communication with job requesters so as to be informed about their job performance and to reduce the risk of potential disputes. Their desire to receive feedback from requesters grows stronger in cases of work rejections because each rejec- tion reduces their chances of attaining Master status. Security entails the provision of assurance, safety, and minimization of work disruptions. Some crowd workers perceived MTurk jobs as secure as they believe that MTurk is unlikely to “go bankrupt.” Nevertheless, lack of job security is a prevalent feeling shared by crowd workers. This is often due to task scamming (i.e., completing work for no payment), which causes disruption in pay and is a potential threat, undermining crowd worker reputations. Crowd workers value account- ability, believing that people’s or institutions’ actions should be traced uniquely and should be accounted for. The current design of the MTurk platform rarely holds job requesters accountable for their behavior, as crowd workers encountered job requesters who abused the CS system (e.g., lying about pay rates). The desire for dignity (sense of pride and respect) is also revealed. Whereas some felt that their work was valued and respected by job requesters, others were disap- pointed in the lack of respect exhibited by job requesters who did not provide an honest worktime estimate or who rejected their completed work unreasonably. For more details on the nine value categories, see Appendix C.

In sum, MTurk mostly fulfills worker values in relation to access, autonomy, and making an impact because the platform offers people from all walks of life free access to work and provides them with control over work decisions, leading to a sense of empowerment. The remaining values were found to be partially, or in some cases, rarely supported, leaving crowd workers feeling exploited. Below we discuss these values and the implications in more detail.

Empowerment Through Value Fulfillment

When a subset of the nine values was implicated in the CS work structures, workers expressed empowerment in relation to (1) meaning, (2) self-determination, (3) impact, and (4) competence. These four types of empowerment are sum-

marized in Table 6, followed by a more nuanced account of each structure.

Empowerment through meaning is a common experience resulting from the value of access to open work opportunities. Access to microtask CS was personally meaningful to 96 per- cent of our respondents. Meaningfulness takes on a number of different forms. For some, access to the CS platform was financially meaningful (making extra income); for others it was cognitively meaningful (feeling productive or mentally challenged); for still others it was experientially meaningful (experiencing enjoyment and excitement).

Further, even when meaningfulness took the same form, such as being financially meaningful, the instantiations of meaningfulness that lead to empowerment were often driven by personal circumstances, as noted by two crowd workers (one currently unemployed and the other owning her business):

It allows me to work from home without wasting time and money driving. Since I’m currently unem- ployed and looking for a job, this allows me to sur- vive in the meantime. (Female, 50 years; Graduate degree; Household income < $25,000; Unemployed)

MTurk has become a secondary source of income that allows me to keep going while I’ve been exper- iencing a downturn in sales. Without MTurk there’s a good chance I might have to abandon my own business and find a “proper” job....it brings in enough money that I can cover my basic living expenses, giving me a little breathing room so I can follow my dream. (Female, 38 years; Some college education; Household income $25,000–$49,999; Employed part-time)

Likewise, the instantiations of cognitive meaningfulness that lead to empowerment were also often driven by personal circumstances. For a 66-year-old retiree, the CS work, coupled with ubiquitous Internet access, allowed her to con- tinue to keep mentally challenged because she needed chal- lenges to keep her mind “in gear.” For a 26-year-old working professional, the nature of a particular CS task (i.e., research HITs) made her “think about the subject matter before responding to the questions asked.”

Feelings of empowerment were also derived from experiential meaningfulness, enjoying the experience of engaging in micro tasks:

I want to be happy. I don’t care anymore about advancement or notoriety or bigger paychecks. I

288 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table 6. Worker Empowerment and Examples

Type Definition* Example Percent**

Meaning The job activities are personally meaningful to me.

“I think that one of the best things [about MTurk] is that it gives me something to do that makes me feel productive, instead of wasting time on the internet.”

96%

Self- determination

I can decide on my own how to go about doing the work.

“As [an MTurk] worker you can choose what you want to do. If you want to do batches one day and surveys the next, it’s up to you. I also think it’s important … that you can work when you have time, at any hour of the day.”

85%

Impact I have a significant influence on others.

“I like that I can contribute to research studies that benefit society. For example, on Saturday I completed a writing HIT that involved describing job opportunities in Florida. I think that helps people … looking for work that guides them to find a better match.”

16%

Competence I am confident about my skills and capabilities to do the work.

“I like that it gives me the opportunity to use some of the skills I learned in college. For example, there’re a handful of jobs that come along that require the ability to do research. I’m quite good at researching and this comes in handy.”

15%

*Definition of the four empowerment cognitions are adopted from Thomas and Velthouse (1990). **Each respondent may have their statements assigned to more than one code so that percentages do not sum to 100%.

worked a six figure job before the cancer and no matter how hard I worked or how much money I made, it never felt fulfilling. (Male, 33 years; Bachelor’s degree; Household income $25,000– $49,999; Employment-Other)

Empowerment through self-determination stems from the value of autonomy in CS work. Crowd workers enjoyed the power of decision-making, such as “having the ability to return HITs.” The free will exerted in work choices is nuanced in terms of what, when, where, and how to work:

I like that I can work anywhere with an Internet connection…I got my hair done last week and the salon has free Wifi. I did a few simple HITs on my iPad while I waited. I also like that tasks are rela- tively brief. I can start and finish tasks while doing other things. (Female, 25 years; Bachelor’s degree; Household income $50,000–$74,999; Unemployed)

Crowd workers’ sense of autonomy is also augmented by CS task characteristics; for example, the variety of MTurk tasks afforded workers opportunities to create a portfolio of preferences:

There are at any time over 200,000 HITs of many different types available....Since [they] take a short time, I can accept them 24 hours a day at my leisure, I can complete work on my computer anywhere…I have an Internet connection, and many HITs allow me to do creative things like write for blogs, make

videos, provide feedback, brainstorm, etc. (Male, 33 years; Bachelor’s degree; Household income $25,000–$49,999; Employment-Other: Disabled)

Empowerment through impact, arising from the value of making an impact on others and on society in general, affords a sense of contributing something for the greater good:

I enjoy being part of a larger purpose and like helping with audio and psychological research. There’re jobs that help make computer speech better for blind people and I feel a sense of accomplish- ment and community when I work on things like that. (Male, 42 years; Bachelor’s degree; Household income $25,000–$49,999; Employed part-time)

While empowerment through impact was valued by only a small proportion (16%) of our sample, we argue that its presence could potentially strengthen the overall sense of empowerment, as illustrated by the following:

I also enjoy doing academic surveys because I learn a lot from them and I’m helping other people with their research….It makes me feel useful when I’m helping others. (Female, 33 years; Bachelor’s degree; Household income $100,000 or more; Employed part-time)

In addition, the feeling of making an impact can also play a moderating role by attenuating the effects of marginalization. Despite feeling competently marginalized, crowd workers

MIS Quarterly Vol. 40 No. 2/June 2016 289

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

continued to do CS work because they feel good about making an impact:

75% of [HITs] are mindless, meaningless, repetitive work that robots will eventually be able to do. The academic surveys are the one area where I feel like I'm contributing to a greater good, so those satisfy me. (Female, 41 years; Graduate degree; Household income $25,000–$49,000; Employed full-time)

Empowerment through competence, ensuing from the value of access to diverse kinds of micro tasks, is valued by a small proportion (15%) of the crowd workers in our sample but is valued nonetheless:

It’s made me more proficient on a computer and has definitely kept my mind active and alert. So…I believe it helps. It also gives me something to fill the gap in my resume from being unemployed. (Female, 50 years; Graduate degree; Household income < $25,000; Unemployed)

Given that micro tasks are often relatively mindless and repetitive, reported instances of empowerment through com- petence may be expected to be smaller. However, their presence, along with the other, more dominant empowerment cognitions of meaning and self-determination, could poten- tially strengthen the overall sense of empowerment:

[CS] is improving my analytical skills and percep- tions. It’s making me aware of political issues and [being more] self-aware. It’s helping me become more articulate and providing real emotional and financial satisfaction. (Female, 44 years; Bachelor’s degree; Household income $25,000–$49,000; Employment-Other)

In sum, our study shows that the four cognitions identified in the extant literature (i.e., Spreitzer 1995) combine to reflect crowd workers’ experiences of empowerment. Moreover, empowerment was experienced by all groups across our sample. To illustrate the point, Table D1 in Appendix D sum- marizes the distribution of empowerment by two demographic factors (employment status and educational level attained). Moreover, our analysis shows that the associations between values and the four cognitions of empowerment vary (Table D2 in Appendix D). For example, all of the workers who valued autonomy felt empowered via self-determination. Although two cognitions (meaning and self-determination) appear more pronounced, we argue that the other two (compe- tence and impact) are important in acting as buffers that mitigate workers’ feelings of marginalization, a topic to which we now turn.

Marginalization Due to Unfulfilled Values

Crowd workers felt marginalized when some of their key values were not supported by the CS platform. The resultant conditions led to a sense of collective powerlessness. Specifically, marginalization manifested in (1) economic marginalization, (2) institutional (policy) marginalization, (3) institutional (technical) marginalization, and (4) com- petence marginalization. Table 7 summarizes these forms of marginalization. Unlike empowerment, there is not always a clear one-to-one mapping between a dimension of marginali- zation and a particular value. Overall, crowd workers felt marginalized when they could not exert their personal agencies to attain the values of fairness, dignity, security, communications, accountability, and transparency.

We found that the most prevalent types of marginalization emerged primarily as a result of economic (60%) and institu- tional policy and practice (38%) concerns that fostered conditions for perceived exploitation. How values are impli- cated (or not) in the CS platform determines the degree of marginalization experienced. Below, we provide a more nuanced account of the marginalization being experienced by organizing the discussion around the four types of marginali- zation when fairness, dignity, security, communication, accountability, and transparency failed to be adequately supported.

Economic marginalization refers to a feeling of being exploited as a result of perceived inequities in MTurk’s com- pensation structure. As noted, MTurk recommends a com- pensation rate of $0.10 per minute. In their responses, crowd workers emphasized that pay rates offered by some job requesters are unfairly low, with some of them not even reaching the recommended minimum:

It’s unfair and inhumane to pay people an average of $2 per hour. I make more than that sometimes, but overall the pay is far too low, and I feel it’s exploitative of people who are desperate to make money. (Female, 41 years; Graduate degree; House- hold income $25,000–$49,999; Employed full-time)

Moreover, crowd workers also raised concerns with regard to job requesters not taking into consideration the nature (com- plexity) of tasks when setting pay rates. For example, writing HITs may require more time and effort than other categories (e.g., tagging HITs). The recommended pay rate across all types of HIT discourages crowd workers who would other- wise be interested in such tasks:

The pay rate is terrible for many jobs, especially writing jobs, and it would be nice to see that change.

290 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table 7. Worker Marginalization and Examples

Type Definition* Example Percent**

Worker Marginalization

Economic marginalization

Feeling exploited The pay can be really atrocious, akin to sweatshop wages. It would be nice if everything had a fair wage. (Female, 34 years; Some college education; Household income $25,000–$49,999; Employment-Other: Stay-at-home Mom)

60%

Institutional (policy) marginalization

Feeling helpless in relation to job requesters and the CS platform

Stop all scammers who put up HITs and don’t pay. We have no recourse at all because what the requester says is final. There’re many requesters who put up HITs who never intend to pay ... If there was a way to stop the scammers, it would make [M]Turk a much safer and more fun place to work. (Male, 49 years; Some college education; Household income < $25,000; Unemployed)

38%

Institutional (technology) marginalization

Feeling constrained by the technical functionalities of the platform

Make a reliable, uniform, automatic system of keeping track of whether I've already participated in a task that doesn’t allow retakes. Keeping track of the tasks I've already done is burdensome especially when requesters change the names of their reposted HITs. (Male, 34 years; Bachelor’s degree; Household income $50,000–$74,999; Employed full-time)

23%

Competence marginalization

Feeling deskilled from doing simple and repetitious work

Sometimes I feel like that I am doing the same types of tasks, or even the same exact task, over and over again. (Male, 19 years; High school graduate; Household income $75,000–$99,999; Employment other)

18%

*The definitions of the four marginalization dimensions are derived from the study. **Each respondent may have their statements assigned to more than one code so that percentages do not sum to 100%.

I’m a writer by trade, and freelance in other areas of MTurk, but I rarely take content jobs here because the pay is insultingly low. (Female, 38 years; Some college education; Household income $25,000– $49,999; Employed part-time)

Institutional (policy) marginalization refers to a feeling of helplessness arising from MTurk’s governance structure, where policies and procedures are perceived to dispropor- tionately favor job requestors. We noted that crowd workers were often on the lookout for MTurk policies and guidelines that would ensure successful transactions with job requesters. As they reflected on their experiences, they voiced how the policies and procedures (or lack thereof) implemented by MTurk engendered feelings of helplessness and power- lessness. Two frequently cited policies relate to the Master qualification and payment rejections.

MTurk confers Master status, following evaluation of workers’ performance, to those who perform well. MTurk defines Master workers as

elite groups of Workers who have demonstrated accuracy on specific types of HITs….Workers achieve a Masters distinction by consistently com- pleting HITs of a certain type with a high degree of

accuracy across a variety of Requesters” (https://requester.mturk.com/help/faq).

Crowd workers seek Master status because it allows them access to better paid HITs. However, certain groups of workers—those who perform content HITs (e.g., writing and survey work)—found themselves being excluded:

I’d also like to see the master’s program become available to more people because…it seems like it’s mainly available to people who do categorization or photo moderation tasks…a lot of requesters for surveys don’t realize this and require a Master’s qualification. It leaves people like me out. (Female, 30 years; Bachelor’s degree; Household income $25,000–$49,999; Employed part-time)

Further, crowd workers feel powerless when their work is rejected; they do not understand requesters’ reasons for rejection but have no way to appeal because “what the requester says is final.” This sense of helplessness is due to two reasons. First, there are no direct channels of commu- nication with job requesters to clarify rejections, and in some cases, job requesters are scammers who plan to take com- pleted work without payment. Second, there exists an asym- metry in the reputation of workers and requesters: workers’

MIS Quarterly Vol. 40 No. 2/June 2016 291

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

reputations are determined by their acceptance rate—a fundamental feature of MTurk. Large proportions of rejected work damage worker reputations and lower their chances of future work. By contrast, there is no systematic reputation mechanism for requesters. Requesters can refuse to pay for any or all work done without reason. There is no appeal pro- cess, and crowd workers have called on MTurk to intervene by implementing policies to keep job requesters accountable and honest.

Institutional (technology) marginalization is a feeling of exclusion when the platform features prevent crowd workers from fully participating. As much as they appreciate the open, convenient access to MTurk, crowd workers point to the inadequacy of some platform features. For example, some find the current technology structure insufficient in (1) facili- tating their communications with job requesters, (2) keeping track of their work history, and (3) protecting them from bad requesters and scammers. When workers wish to clarify a HIT requirement or seek answers to HIT rejections, they find MTurk’s technical features lacking:

I’d also like if requestors made it easier to commu- nicate with them. Sometimes I have a question and a requestor doesn't bother to respond. Sometimes I’ve had to return surveys I’ve spent a lot of time on…or there has been an error in the study and I’m unsure what to do next. I wish the time could be longer sometimes also. I’ve had HIT's expire on me even when working at a good pace. (Female, 33 years; Bachelor’s degree; Household income $100,000 or more; Employed part-time)

Crowd workers also find it difficult to keep track of HITs they perform. This is unsurprising as, on average, those surveyed complete almost a thousand (958, to be precise) HITs each week. Survey respondents noted that the current MTurk plat- form does not help them account for work performed. To make matters worse, if the same HITs are repeated uninten- tionally, they will receive a rejection, negatively affecting their work statistics:

The biggest problem I have…is the inability to find out quickly and easily if I’ve done the HIT pre- viously! Out of the 20 rejections that I have…I’d say 17 of them are for attempting to do a HIT that I’d previously done!…this is an honest mistake… but it still counts against our average; [MTurk] need[s] to come up with something on the dashboard that will allow us to quickly check past work. (Male, 45 years; Associate degree; Household income $25,000–$49,999; Employed full-time)

Further, some workers complain that the current system offers only limited tools/mechanisms to allow them to scan, filter, or block bad requesters and scammers. The increasing presence of scammers gives rise to workers’ feelings of vulnerability:

I dislike the lack of resources for workers to com- plain about scammers who repeatedly post HITs trying to dupe people into disclosing financial infor- mation, click advertisements, or sign up for credit cards. If there was a better way to report and block those types of jobs, I think it would be great. (Male, 30 years; Bachelor’s degree; Household income $75,000–$99,999; Employed part-time)

However, oftentimes the implementation of technical features is closely related to MTurk’s governance structure. For example, with regard to the implementation of clearer and stricter rules on rejecting a completed HIT, some workers urge MTurk to allow job requesters more options for HIT rejection, proposing technical functionality to distinguish between declining and rejecting completed work: a decline decision would not jeopardize a worker’s standing while a reject decision would:

I’d like to see a system set up that would allow a requester to decide not to accept work without actually rejecting the worker. If you do a writing task…in good faith…you should not end up with a rejection on your permanent record that affects your ability to work….[R]andom rejections discourage many people from even attempting these HITs. Amazon should probably make more effort to penalize or ban requesters [who] steal writing by publishing work they rejected. (Female, 38 years; Some college education; Household income $25,000–$49,999; Employed part-time)

Competence marginalization refers to becoming deskilled as a result of doing simple, repetitive work on MTurk. Given the preponderance of simple, mundane tasks, some workers called for greater variety:

I’d like to see more [computer coding] HITs. I’m a programmer and would love to put my skills to work....Most of [the] HITs pay so little it’s hardly worth doing them. Doing a lot of HITs does in- crease your [income], so that's what’s driving me to do [these] garbage HITs. (Male, 31years; Some college education; Household income < $25,000; Unemployed)

Similar to empowerment, marginalization was experienced by all groups in our sample. Table E1 in Appendix E sum-

292 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

marizes the distribution of the four types of marginalization by two demographic factors (employment status and educa- tional level). Revealing exclusive and distinct associations between the nine values and the four cognitions of marginali- zation is difficult given the associative nature of crowd workers’ values. However, the distribution of values along the four cognitions of marginalization (Table E2) seems to suggest certain patterns. For example, a majority of those who value fairness and dignity often expressed a sense of economic marginalization, while a majority of those who value transparency and accountability often felt marginalized by institutional policies. These patterns could be examined in future work to reveal the variability in the associations between values and the four cognitions of marginalization.

In summary, survey respondents revealed multiple values in relation to their engagement in CS. Yet, the extent to which their expected values are fulfilled varied considerably. In instances when a value was fulfilled, crowd workers felt em- powered; otherwise, they felt powerless and even exploited. Through this interplay between the values and dimensions of empowerment and marginalization emerges the duality implicit in CS, and this is discussed next.

The Duality of Microtask Crowdsourcing: Structures of Empowerment and Marginalization

The duality is intrinsic to the lived experiences of the crowd workers.9 The duality of microtask CS is reflected in two contrasting yet coexisting feelings of empowerment and marginalization experienced simultaneously by the same crowd workers. These dual experiences surface when crowd workers interact with CS work structures that mediate their activities. The four structures of microtask CS are compensa- tion, task, governance, and technology (as detailed in the “Microtask Crowdsouring” section of this paper). As a medium of crowd workers’ conduct of work, these structures have both empowering and marginalizing implications on crowd worker activities. They feel empowered when the structures enable choice (e.g., where and when to work satisfying the value of autonomy); they feel marginalized when the same structures restrict action (e.g., lack of commu- nication channels limiting their opportunities to voice con- cerns). In effect, the dual experiences are collectively

determined by the extent to which the nine values are undermined or promoted in the design of these four work structures, as illustrated in Figure 1.

IS literature recognizes the dual outcomes of ICT (see Markus 2014). The paradoxical nature of ICT, where it simul- taneously produces opposite effects, has been found in a wide range of contexts, such as organizational structures (ibid.), individual users (Jarvenpaa and Lang 2005), and communities (Harris and Weiner 1998). There is thus a need to account for the contradictory consequences of ICT (Markus 2014; Robey and Boudreau 1999). The duality of microtask crowdsourcing accounts for such contradictions by highlighting that, while the use of Internet technologies to create new work structures that are largely adhocratic and less bureaucratic can provide workers with a sense of control and freedom that is em- powering, current work structures that are baked into the CS infrastructure privilege requestors in ways that undermine crowd workers’ sense of empowerment.

The four microtask CS structures at times individually and sometimes collectively conspire to engender opposite feelings of empowerment and marginalization which are unequal and nonuniform. The structures of compensation and governance evoke stronger feelings of marginalization while the task and technology structures are largely responsible for engendering the feeling of empowerment. For instance, on one hand, the feeling of marginalization is deeply rooted in the current microtask compensation structure where the extremely low payment rate results in feeling being exploited. On the other hand, the empowerment derived from free, open, and equal access to earn a supplemental or regular income is quickly and easily provisioned through this very structure. Similarly, the governance structure is perceived to favor requestors because they can control task rate and payment rejection without explanation while the crowd workers have no bar- gaining power except the “take-it or leave-it” option, leaving them with a feeling of helplessness. Conversely, this take-it or leave-it option provisions popular work pattern flexibility where workers can finish the tasks they want to and return those they no longer wish to complete. Nonetheless, our analysis suggests that the governance structures are currently designed mostly to reinforce traditional regimes of managerial power and control (Kraemer and Dutton 1979; Markus 2014; Pinsonneault and Kraemer 1997) over the workforce, possibly a conscious effort by MTurk to mitigate requestors’ risks as they are the source of MTurk’s revenues. Meanwhile, the CS platform owners who provision the technology structure assume the role of dispassionate technology supplier of a free, open marketplace where there are no barriers to “becoming” a worker, but simultaneously fail to provide adequate techno- logical tools and functions to meet workers’ communication and microtask management needs. Similarly, the impersonal

9Among the 210 workers surveyed, 70% (147 workers) simultaneously experienced at least one form of empowerment and one form of margin- alization. The duality experienced by the majority of those surveyed is distributed relatively evenly across the demographic factors of gender, education level, employment status, and household income (Appendix F).

MIS Quarterly Vol. 40 No. 2/June 2016 293

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Empowerment

Marginalization

Meaning Self-determination Impact Competence

Economic Institutional (policy) Institutional (technical) Competency

Figure 1. The Duality of Empowerment and Marginalization in Microtask Crowdsourcing

programmed coordination (Finnegan and Longaigh 2002; Markus 2014) implicated in the microtask structures provision the design of repetitive and routinized tasks that allow for work pattern flexibility, but simultaneously discourage workers from continuing to engage in deskilled tasks because they are dehumanizing and lack opportunities for skill development.

Duality is a critical property of this phenomenon. By uncov- ering the presence of duality, we create a space for examining and interpreting opposing feelings of empowerment and mar- ginalization that are simultaneously voiced by the same crowd workers. The emergence of such a space pushes the CS dis- course from the silos, where it is either primarily touted as the wellsprings of entrepreneurial creativity (e.g., Greengard 2011; Kaganer et al. 2013) or mostly portrayed as the har- binger of digital sweatshops (e.g., Harris 2011; Marvit 2014), to a more dialogic discourse where these two views are not mutually exclusive but rather mutually constitutive (Schultze and Stabell 2004).

Our analysis of the duality suggests that, in order to reap the benefits of microtask CS, such intricacies as those associated with the underlying structures of empowerment and margin-

alization should be well understood and carefully considered in designing CS platforms and policies. In addition to pro- viding confirmatory evidence in support of contradictory consequences of ICT, the current study makes a number of novel and important contributions to and for IS research, as well as raising some crucial ethical issues.

Contributions to and for IS Research

To begin with the contributions to IS research, our study contributes to both CS and VSD research. Three contribu- tions are made in relation to CS research. First, and on the positive side, by identifying the sources of crowd worker empowerment, our research reveals those critical techno- logical features that support microtask CS and that drive worker engagement. Strong feelings of empowerment arise from open access to job opportunities and work pattern autonomy. We argue that human values of open access to jobs and work pattern autonomy provisioned by CS tech- nologies transcend all types of work and could help empower workers in other forms of CS, but only if issues of marginali- zation are confronted and ameliorated.

294 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Thus, second, and much more negatively, by exposing sources of worker marginalization rooted in the conduct of microtask CS, our study reveals strong feelings of worker marginalization arising from incongruities between worker values of fairness and the lack of fair compensation and governance practices. Worker marginalization could not only jeopardize the future of CS but also raise crucial ethical concerns that we argue simply must be addressed. Current practices ostracize workers who could otherwise benefit from CS work. Prior studies view CS as a model where organi- zations use Internet technologies to harness the efforts of crowd workers to perform organizational tasks (e.g., Brabham 2012; Saxton et al. 2013). However, what is missing here and in the microtask CS marketplace is appropriate governance of the sourcing contract (e.g., contract negotiation, monitoring, delivery, and closing) and a collective bargaining unit for crowd workers. In our study, crowd workers voiced frustra- tion over extremely low pay, increased scamming, and lack of channels to report and appeal unfair practices, signaling an unregulated online labor marketplace. Prior studies (e.g., Brabham 2012; Kittur et al. 2013; Silberman et al. 2010; Williamson 2014) have raised concerns about the lack of ethical standards in microtask CS. Our study provides further insight into this lack by revealing four types of worker marginalization and uncovering their sources, which are deeply rooted in current microtask CS structures. We shall return to this issue below and in our concluding remarks and reflections.

Third, our inductive findings can be used to explain the interplay among the human values, cognitions of empower- ment, and dimensions of marginalization, thus laying a foundation for future deductive work. Our study reveals a comprehensive list of nine values important to crowd workers in undertaking microtask CS. In particular, a majority of those workers who value accountability, communication, security, and transparency felt that microtask CS allowed them to exert control and to become self-determined, leading to a sense of empowerment. However, the same group of workers expressed feelings of being taken advantage of by the lack of appropriate governance mechanisms, leading to a simultaneous sense of marginalization—the duality to which we refer. Future studies can investigate (the strength of) the relationship between worker values and the CS experience (empowerment vis-à-vis marginalization) by examining the degree to which worker values are implicated in microtask and other CS structures. Such studies could uncover the relative importance of various values, which could be useful in understanding how crowd workers make tradeoffs to resolve conflicting values and which could inform improve- ments to current CS-related practices and policies. An associated avenue for future CS research is to focus on the

microtask structure (i.e., increasing task significance via task structure can empower workers through meaning and impact). While our study confirms the positive role of Internet technologies in affording open access and individual freedom to crowd workers, it also reveals that deficiencies embedded in the technology structure of CS platforms lead to worker marginalization. Future studies could test the relationship between worker values (e.g., communication, transparency, accountability) and the underlying CS structures (e.g., governance, technology). Insights from such studies could not only help address the criticism that “[t]hese technologies are not enabling people to meet their potential; they’re instead exploiting people” (Cherry, cited in Marvit 2014) but could— and should—also have far reaching consequences for job requesters and platform providers, as well as for the IS research community itself.

With respect to VSD research, our study contributes in two ways. First, it contributes by empirically identifying an additional set of values within the CS context. Our approach to the study of crowd worker values is consistent with calls by VSD researchers (e.g., Le Dantec et al. 2009) who recom- mend moving away from largely fixed value classifications to a more flexible value repository that accounts for context. Our study reveals positive values (autonomy, dignity, fair- ness, security, transparency, and accountability), which are consistent with Friedman and Kahn’s (2003) perspective of the universality of human values, as well as such values speci- fic to the CS context (i.e., access; communication, making an impact). Additionally, we uncover aspects of CS that fly in the face of ethical design. We, thus, extend previous research by identifying a novel set of human values implicated in the CS context, recognizing that different patterns of values might emerge from different contexts. These CS-centric values with moral epistemic standing should serve to ethically ground the design and development of CS platforms. Our study thus posits an initial repository of human values with ethical import for CS platform design. This repository can be used as a foundation for classifying values that can be extended by future design science researchers through an iterative and integrative process defined in VSD’s tripartite investigations involving conceptual, empirical, and technical analysis. Second, we broaden VSD’s reach by applying it in a more complex social–technical system (STS) than is usual in VSD studies, which focus primarily on the use of micro computa- tional systems such as Web browsers, groupware, simulation system, and RFID (e.g., Friedman et al. 2008; Millett et al. 2001). Extending the application of VSD to the CS context is challenging because VSD prescriptions are not necessarily readily applicable. However, the approach used in this study offers an initial illustration of how VSD may be used to con- duct the empirical portion of VSD’s tripartite investigations.

MIS Quarterly Vol. 40 No. 2/June 2016 295

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Our study also has profound implications for IS researchers and other communities that undertake research utilizing CS platforms (e.g., to gather and/or analyze data). Picking up on themes raised by, inter alia, Desouza and his colleagues (2007; 2006), we argue that we as a community of IS scholars simply cannot stand by and accept the marginalization and exploitation of crowd workers. As well as arguing for ethical design in CS systems (work processes as well as techno- logical platforms), we argue for ethical design of our own research practices, for example, in relation to providing appropriate compensation to crowd workers for their contribu- tions to research studies, and in advocating for appropriate research policies on the part of our universities.10 We return to this theme in our concluding remarks.

Implications for Design Science Researchers

The design science paradigm is gaining popularity within IS research internationally (Hevner et al. 2004).11 VSD provides a comprehensive framework for advancing a value-centered research and design agenda, which is different from user- centered design perspectives (Le Dantec et al. 2009). VSD scholars distinguish between usability and human values with ethical import by arguing that usability refers to system properties that make it work in a functional sense; however, in this sense, usability does not guarantee the support for ethical values (Friedman et al. 2008). The VSD approach can prove useful in bridging the gap between efficient design of sociotechnical systems (STS) and ethics by prioritizing consideration of human values with ethical import in design. To this end, the implications of this work for design science researchers are twofold.

First, the approach adopted in this study provides an ethical design perspective that can be used in future research to analyze societal problems entangled in STS. Such an ap- proach can be useful in eliciting values through a harm and benefits analysis (Freidman et al. 2008) during requirements gathering. The derived values can be used to conduct value- sensitive analysis to evaluate the consequences of impli- cating—or not implicating—the values in new systems. Incorporating a value sensitivity component in requirements gathering to infuse features that embody human values of

ethical import could help in mitigating system failures (Friedman et al. 2006).

Second, we offer an agenda for future design science research that could help advance ethical considerations in the design of microtask CS work systems by being mindful of how crowd workers’ values are implicated in CS structures to produce dual effects. Design investigations that are sensitive to crowd worker values are more likely to produce designs that provi- sion worker empowerment while reducing worker marginali- zation. This in turn could foster a motivating environment where crowd workers feel appreciated rather than resentful. We posit that a more appreciative workforce is likely to be more productive and less likely to collectively protest against the interests of job creators. A greater balance of power pro- vides fewer possibilities for abuse, allowing microtask CS to develop into a more open and free job marketplace, a prin- ciple germane to the sustainability of this kind of CS. Specifi- cally, we propose two kinds of analysis that can be conducted to further this work. One is value-driven investigation on microtask CS platforms where a more detailed and nuanced tripartite analysis is done on each value (see Table G1 in Appendix G); the other is a similar, structure-driven analysis that focuses on the four structures where the values are implicated (see Table G2 in Appendix G).

Implications for Practitioners

Our analysis suggests that understanding the salient values is instrumental in affecting crowd workers’ sense of empower- ment and/or marginalization. In this regard, our study offers several useful guidelines to the three key practitioner stake- holders in microtask CS: crowd workers, job requesters, and platform designers and owners.

Guidelines for Crowd Workers

We argue that crowd workers should collectively mobilize to ensure that their voices are heard and not drowned out when the drumbeat for regulating the on-demand workforce grows louder (O’Donovan 2015). Crowd workers can collectively voice their concerns to affect change by participating in such forums as DYNAMO (http://www.wearedynamo.org/), Turkopticon (https://turkopticon.ucsd.edu/), and MTurk Forum (http://mturkforum.com/). They are not obliged to accept marginalization just because some aspects of microtask CS empower them. The dual experiences of marginalization and empowerment are not mutually exclusive and can coexist; thus, one can express discontent with one (e.g., voice con- cerns over unfair payment practices) without the fear of losing the other (e.g., open access to work on the MTurk platform).

10In relation, for example to IRB policies (see http://wiki.wearedynamo.org/ index.php/Guidelines_for_Academic_Requesters).

11See, for example, Stein et al. (2014) for evidence of the growth of design science research in the European IS academy.

296 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Where walking away from this work (when one feels they are being exploited) is not an option, then workers are being marginalized and have the right to voice their concerns.

Guidelines for Job Requesters

The crowd workers in our sample would be considerably more motivated were job requesters to carefully evaluate their rates of payment and alternative microtask designs. Our analysis suggests that crowd workers appreciate CS jobs that allow them to make contributions to scientific research and societal initiatives. However, they feel strongly about being exploited because of ambiguous task instructions and unfair compensation. A requirement for job requesters, would be to provide clearer task instructions and more accurate estimates of time requirements to allow crowd workers to make informed decisions in accepting jobs. Additionally, our data suggest that job requesters should make an effort to maintain open communication with crowd workers, providing them with feedback on work undertaken so that they feel more appreciated. Again, this could usefully be incorporated into CS policies to facilitate more ethical practices.

As for academic researchers, it is clear that CS platforms such as MTurk have been increasingly used by researchers (job requesters) from different disciplines for field experiments (e.g., Alonso and Mizzaro 2012; Chandler and Kapelner 2013; Mason and Suri 2012). As IS researchers, we should be keenly aware of the ethical considerations of employing microtask CS methods in our own research by being mindful of the value of crowd worker; for example, by applying an appropriate rate of compensation without hiding behind the guise of poor research financial support available at many academic institutions, by keeping communication channels open, and by explaining how the crowd workers are making an impact by participating in a research study, as we have (somewhat belatedly, we have to admit12) attempted to do in this study. Academic researchers have to ask themselves: Is their scientific inquiry worth investigation if it is done on the backs of crowd workers who are reluctantly accepting low paying HITs because they have no other opportunities for employment?

Guidelines for CS Platform Designers and Owners

Designers and owners of microtask CS platforms are advised to listen to crowd workers with a view to improving platform

design and updating technical functionalities. CS workers appreciate access to job opportunities, the freedom to make their own job decisions, and the opportunity to integrate work and family obligations. However, they express dissatisfaction with inappropriately low levels of compensation for their work, the limited communication mechanisms, vague stan- dards and policies, and the lack of intervention by platform owners in penalizing fraudulent job requesters. We thus argue that CS platform owners should introduce policies and practices that require job requesters to recompense crowd workers at rates that are appropriate no matter where in the world the worker is located. Platform owners should take their responsibilities seriously and no longer tolerate inappro- priate payment or fraudulent behavior on the part of job requestors. Specifically, we strongly recommend that Amazon implement the following design and policy changes to their MTurk platform:

1. Set fair minimum payment rate: Set a minimum hourly rate for all MTurk workers that is fair. MTurk has a minimum commission rate that they take from the requestors; similarly, they can devise a minimum hourly rate that would mitigate unfair payment practices. We suggest that Amazon consider crowd workers’ slack time between two tasks and set a rate to offset the time lost between tasks.

2. Stop the scams: Protect crowd workers by taking action against the requestors who engage in fraudulent and harmful activities. Have clear policies regarding what is considered scamming and what are the consequences of engaging in such scamming activities. Create a formal governance structure that can be used to identify and punish the scammers. In addition to investigating workers’ scam complaints, allow them to block the requestors.

3. Make Master level qualification objective and trans- parent: The experience of the crowd workers who parti- cipated in our study suggests that attaining master level qualification is at best difficult to understand and at worst awarded on MTurk’s whim. MTurk should conscien- tiously adhere to the policy they have posted on their platform regarding the qualifications required for at- taining the master level designation. In addition, MTurk should design platform functionalities that would allow workers to track their progress toward master level; the pathway to master level should not be obscure nor ad hoc.

4. Establish channels for open communication: Here we make three recommendations. First, implement a formal mediation process that is run by MTurk through which12Please refer to our further reflections in the following section.

MIS Quarterly Vol. 40 No. 2/June 2016 297

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

the workers can file an appeal and achieve resolution on conflicts arising from unfair treatment. Second, create a requirement where the requestors have to provide an explanation when they reject work or block a worker. Requestors should not be allowed to deny a payment on a completed task or block a worker without good reason. Third, provide communication mechanisms (e.g., instant messaging; online chat) that workers can use to directly communicate with the requestor using the platform features.

5. Update the platform: The platform, since its inception, has been in the Beta version. The platform features need to be updated to better support the crowd workers who currently spend time searching for HITs for which they are not paid. They also spend time installing multiple outside scripts and extensions to work effectively on MTurk for with they are not paid. They have to keep track of their work histories for which they are not paid. They have to deal with work rejections for which they are not paid. MTurk needs to take ownership of the role it plays in the CS marketplace and design a work plat- form that works for the crowd workers.

6. Enforce polices uniformly and transparently: Enforce the stated terms of service by taking action when requestors violate those policies. Enforce the policies consistently and inform the workers and/or the requestors when the policy is being violated by explaining how and why this is the case. Sending automated and/or canned e-mails do not suffice when workers’ potential to gain future work is at stake.

Concluding Remarks and Further Reflections

We as a society have reached a point of social consciousness where the debate regarding the need for new policies to address the effects of the emerging “gig” or on-demand economy on its workers is taking center stage (e.g., Shaban 2015). We hope that our study contributes to this debate by highlighting that CS enables human beings to achieve autonomy and independence but it does not serve as a panacea for promoting all human values expressed by the crowd nor that should be expected by society. To achieve a healthier and more sustainable CS work environment, we need to pay greater attention to the institutional practices and societal impacts of microtask CS. Our study raises serious concerns about the risks emerging from the disparities of power in microtask CS and calls for more attention to be paid to mitigate worker marginalization. The duality of empower-

ment and marginalization has implications for participants in the MTurk marketplace and could well be applied in other microtask CS contexts (e.g., MobileWorks; Crowdflower; CloudFactory). Our study relies on data collected via the MTurk platform and has given voice to the crowd workers; future research could usefully consider their work in situ, giving greater emphasis to observations of actual practices (e.g., Whittington 2014). Moreover, our study is restricted to MTurk workers based in the United States to reduce any confounding factors arising from different countries and economies. Further research on international comparisons of microtask CS will enhance our understanding across cultures and national boundaries.

We ourselves have learned much from our study. We com- menced our study with certain assumptions about the nature of the CS workforce (e.g., in terms of supplemental income and reasonable compensation levels), only to find that these assumptions were naïve at best. We found that many crowd workers are totally dependent on income arising from their microtask CS work, and our own research design inadver- tently lead to inadequate recompense for the detailed responses we received from those who contributed to the study. We have subsequently provided a bonus to the crowd workers and raised the payment rate to the equivalent of a $15 hourly wage. This is similar to the approach adopted by Williamson (2014) in retroactively raising the payment to her MTurk survey respondents.

Limitations notwithstanding, our study presents a deep analysis of crowd worker values in what is an increasingly prevalent social phenomenon. While studies on CS platforms, technical systems, and job requesters are informative, our understanding of CS cannot be complete without an under- standing of the crowd itself. CS is only possible with the active enrollment of individual workers. This article offers initial steps in depicting the duality manifested in the technology-enabled marketplace that is CS. Clearly, this particular ICT has the potential to augment the voice of the less powerful—the crowd workers—through interactive com- munication channels and efficient job filtering. However, by no means all the values that have been voiced can be sufficiently afforded by technology alone. Crowd workers’ reasonable desires for fair compensation and just procedures can only be met by implementing governance policies on compensation and labor relations. A comprehensive under- standing of the microtask CS phenomenon will benefit from a multidisciplinary approach that goes beyond ICT (von Krogh and Spaeth 2007). More research is clearly needed to develop technological solutions and governance mechanisms that facilitate improved CS employer–employee communi- cation and to ensure fairness and transparency if we are to take up challenges by the likes of Mumford (1981) and Desouza and his colleagues (2007; 2006).

298 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

In the long term, assessing societal impacts of Internet-based technologies and the growing crowd worker population are emerging as active and fruitful areas for researchers and policy makers alike. By seeking to understand the values revealed by crowd workers, we trust that this study will heighten awareness of worker marginalization and lay a foundation for developing and implementing CS platforms and governance mechanisms that empower the crowd work- force, with a view to creating an equitable and rewarding work environment. As a research community, we have a responsibility not only to uncover what is ethical but also to act on our findings. Depending on how well values sensitive to crowd workers are implicated in CS technical platforms and practices could determine whether this increasingly prevalent form of work is a harbinger of worker emancipation or exploitation.

Acknowledgments

We would like to thank M. Lynne Markus and all the members of the review team for their helpful and supportive comments and positive critiques of previous versions of this paper throughout the whole review process. We have gained much from their insight.

References

Alonso, O., Mizzaro, S. 2012. “Using Crowdsourcing for TREC Relevance Assessment,” Information Processing & Management (48:6), pp. 1053-1066.

Berinsky, A. J., Huber, G. A., and Lenz, G. S. 2012. “Evaluating Online Labor Markets for Experimental Research: Amazon. com’s Mechanical Turk,” Political Analysis (20:3), pp. 351-368.

Brabham, D. C. 2012. “The Myth of Amateur Crowds: A Critical Discourse Analysis of Crowdsourcing Coverage,” Information, Communication and Society (15:3), pp. 394-410.

Bratvold, D. 2011. “Enterprise Crowdsourcing Blasts Off as Social Media Growth Industry” (http://www.businessesgrow.com/2011/ 12/13/enterprise-crowdsourcing-blasts-off-as-social-media- growth-industry/; retrieved October 2014).

Chandler, D., and Kapelner, A. 2013. “Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets,” Journal of Economic Behavior and Organization (90), pp. 123-133.

Chandler, J., Paolacci, G., and Mueller, P. 2013. “Risks and Rewards of Crowdsourcing Marketplaces,” in Handbook of Human Computation, New York: Springer, pp. 377-392.

Deng, X., and Joshi, K. D. 2013. “Is Crowdsourcing a Source of Worker Empowerment or Exploitation? Understanding Crowd Workers’ Perceptions of Crowdsourcing Career,” in Proceedings of 34th International Conferences on Information Systems, Milan, Italy.

Desouza, K. C., Ein-Dor, P., McCubbrey, D. J., Galliers, R. D., Myers, M. D., and Watson, R. T. 2007. “Social Activism in Information Systems Research: Making the World a Better

Place,” Communications of the Association for Information Systems (19), pp. 261-277.

Desouza, K. C., El Sawy, O. A., Galliers, R. D., Loebbecke, C., and Watson, R. T. 2006. “Beyond Rigor and Relevance Towards Responsibility and Reverberation: Information Systems Research that Really Matters,” Communications of the Asso- ciation for Information Systems (17:16), pp. 2-26.

Felstiner, A. 2011. “Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry,” Berkeley Journal of Employment and Labor Law (32:1), pp. 143-202.

Finnegan, P., and Longaigh, S. N. 2002. “Examining the Effects of Information Technology on Control and Coordination Relation- ships: An Exploratory Study in Subsidiaries of Pan-National Corporations,” Journal of Information Technology (17:3), pp. 149-163.

Finnerty, A., Kucherbaev, P., Tranquillini, S., and Convertino, G. 2013. “Keep it Simple: Reward and Task Design in Crowd- sourcing,” in Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI, New York: Association for Computing Machinery, Article 14.

Friedman, B. 1996. “Value-Sensitive Design,” Interactions (3:6), pp. 16-23.

Friedman, B., Howe, D. C., and Felten, E. 2002. “Informed Consent in the Mozilla Browser: Implementing Value-Sensitive Design,” in Proceedings of the 35th Annual Hawaii International Conference on System Science, Los Alamitos, CA: IEEE Computer Society.

Friedman, B., and Kahn Jr., P. H. 1992. “Human Agency and Responsible Computing: Implications for Computer System Design, “Journal of Systems Software (17), pp. 7-14.

Friedman, B., and Kahn Jr., P. H. 2003. “Human Values, Ethics, and Design,” in The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applica- tions, J. A. Jacko and A. Sears (eds.), Mahwah, NJ: Lawrence Erlbaum Associates, Inc., pp. 1177-1201.

Friedman, B., Kahn Jr., P. H., and Borning, A. 2006. “Value Sensitive Design and Information Systems,” in Human– Computer Interaction in Management Information Systems: Foundations, P. Zhang and D. Galletta (eds.), Armonk, NY: M. E. Sharpe, pp. 348-372.

Friedman, B., Kahn Jr., P. H., and Borning, A. 2008. “Value Sensitive Design and Information Systems,” in The Handbook of Information and Computer Ethics, K. Himma and H. Tavani (eds.), Hoboken, NJ: Wiley, pp. 69-102.

Galliers, R. D., and Land, F. F. 1987. “Choosing an Appropriate Information Systems Research Methodology,” Communications of the ACM (30:11), pp. 900-902.

Geiger, D., and Schader, M. 2014. “Personalized Task Recom- mendation in Crowdsourcing Information Systems—Current State of the Art,” Decision Support Systems (65), pp. 3-16.

Goodman, J. K., Cryder, C. E., and Cheema, A. 2013. “Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples,” Journal of Behavioral Decision Making (26:3), pp. 213-224

Greengard, S. 2011. “Following the Crowd,” Communications of the ACM (54:2), pp. 20-22.

MIS Quarterly Vol. 40 No. 2/June 2016 299

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Harris, C. G. 2011. “Dirty Deeds Done Dirt Cheap: A Darker Side to Crowdsourcing,” in Proceedings of IEEE 3rd International Conference on Social Computing, Los Alamitos, CA: IEEE Computer Society, pp. 1314-1317.

Harris, T., and Weiner, D. 1998. “Empowerment, Marginalization, and ‘Community-Integrated’ GIS,” Cartography and Geographic Information Systems (25:2), pp.67-76.

Hevner, A., March, S., Park, J., and Ram, S. 2004. “Design Science in Information Systems Research,” MIS Quarterly (28:1), pp. 75-105.

Horton, J. J., Rand, D. G., and Zeckhauser, R. J. 2011. “The Online Laboratory: Conducting Experiments in a Real Labor Market,” Experimental Economics (14:3), pp. 399-425.

Howe, J. 2006. “The Rise of Crowdsourcing,” Wired (14:6), pp. 1-4.

Howe, J. 2008. “Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business,” The International Achievement Institute.

Jarvenpaa, S. L., and Lang, K. R. 2005. “Managing the Paradoxes of Mobile Technology,” Information Systems Management (22:4), pp. 7-23.

Kaganer, E., Carmel, E., Hirschheim, R., and Olsen, T. 2013 “Managing the Human Cloud,” MIT Sloan Management Review (54:2), pp. 22-32.

Kajino, H., Arai, H., and Kashima, H. 2014. “Preserving Worker Privacy in Crowdsourcing,” Data Mining and Knowledge Discovery (28:5-6), pp. 1214-1335.

Kapelner, A., and Chanler, D. 2010. “Preventing Satisfaction in Online Surveys: A ‘Kapcha’ to Ensure Higher Quality Data,” Crowd Conference, October 4, 2010, San Francisco, CA.

Kittur, A., Nickerson, J. V., Bernstein, M. S., Gerber, E. M., Shaw, A., Zimmerman, J., Lease, M., and Horton, J. J. 2013. “The Future of Crowd Work,” in Proceedings of the 16th ACM Conference on Computer Supported Cooperative Work, January 2013, San Antonio, Texas.

Klein, H., and Myers, M. 1999. “A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Informa- tion Systems,” MIS Quarterly (23:1), pp. 67-94.

Kraemer, K. L., and Dutton, W. H. 1979. “The Interests Served by Technological Reform: The Case of Computing,” Administration & Society (11:1), pp. 80-106.

Le Dantec, C. A., and Edwards, W. K. 2008. “Designs on Dignity: Perceptions of Technology among the Homeless,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York: ACM.

Le Dantec, C. A., Poole, E. S., and Wyche, S. P. 2009. “Values as Lived Experience: Evolving Value Sensitive Design in Support of Value Discovery,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York: ACM, pp. 1141-1150.

Lee, A. S., and Baskerville, R. L. 2003. “Generalizing Generaliza- bility in Information System Research,” Information Systems Research (14:3), pp. 221-243.

Markus, M. L. 2014. “Information Technology and Organizational Structure” in Computing Handbook: Information Systems and Information Technology, Volume II, H. Topi and A. Tucker (eds.), London: Chapman and Hall, CRC Press, pp. 67:1-22.

Markus, M. L., Majchrzak, A., and Gasser, L. 2002. “A Design Theory for Systems that Support Emergent Knowledge Processes,” MIS Quarterly (26:3), pp. 179-212.

Marvit, M. Z. 2014. “How Crowdworkers Became the Ghosts in the Digital Machine,” The Nation, February 4, 2014.

Mason, W., and Suri, S. 2012. “Conducting Behavioral Research on Amazon’s Mechanical Turk,” Behavior Research Methods (44:1), pp. 1-23.

Miles, M. B., and Huberman, A. M. 1994. Qualitative Data Analysis: An Expanded Sourcebook (2nd ed.), Thousand Oaks, CA: Sage Publications.

Millett, L. I., Friedman, B., and Felte, E. 2001. “Cookies and Web Browser Design: Toward Realizing Informed Consent Online,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York: ACM.

Moussawi, S., and Koufaris, M. 2013. “The Crowd on the Assembly Line: Designing Tasks for a Better Crowdsourcing Experience,” in Proceedings of 34th International Conferences on Information Systems, Milan, Italy.

Mumford, E. 1981. “Participative Systems Design: Structure and Method,” Systems, Objectives, Solutions (1:1), pp. 5-19.

O’Donovan, C. 2015. “What a New Class of Worker Could Mean for the Future of Labor,” BuzzFeed News, June 18 (http://www. buzzfeed.com/carolineodonovan/meet-the-new-worker-same-as- the-old-worker#.rdMQ85Jyn; accessed August 8, 2015).

Orlikowski, W. J., and Baroudi, J. J. 1991. “Studying Information Technology in Organizations: Research Approaches and Assumptions,” Information Systems Research (2:1), pp. 1-28.

Pinsonneault, A., and Kraemer, K. L. 1997. “Middle Management Downsizing: An Empirical Investigation of the Impact of Infor- mation Technology,” Management Science (43:5), pp. 659-679.

Robey, D., and Boudreau, M. C. 1999. “Accounting for the Contra- dictory Organizational Consequences of Information Tech- nology: Theoretical Directions and Methodological Implica- tions,” Information Systems Research (10:2), pp. 167-185.

Rosenberg, T. 2011. “Crowdsourcing a Better World,” New York Times, Opinion Pages, March 28 (http://opinionator.blogs. nytimes.com/2011/03/28/crowdsourcing-a-better-world/).

Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., and Tomlinson, B. 2010. “Who Are the Crowdworkers? Shifting Demographics in Mechanical Turk,” in CHI’10 Extended Abstracts on Human Factors in Computing Systems, New York: ACM, pp. 2863-2872.

Ryan, G. W., and Bernard, H. R. 2000. “Data Management and Analysis Methods,” in Handbook of Qualitative Research, N. Denzin, and Y. Lincoln (eds.), Thousand Oaks, CA: Sage Publications, pp. 769-802.

Saito, S., Watanabe, T., Kobayashi, M., and Takagi, H. 2014. “Skill Development Framework for Micro-Tasking,” in Universal Access in Human–Computer Interaction: Universal Access to Information and Knowledge (Volume 6, Part II), C. Stephanidis and M. Antona (eds.), New York: Springer International Publishing, pp. 400-409.

Satzger, B., Psaier, H., Schall, D., and Dustdar, S. 2013. “Auction- Based Crowdsourcing Supporting Skill Management,” Information Systems (38:4), pp. 547-560.

300 MIS Quarterly Vol. 40 No. 2/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Saxton, G. D., Oh, O., and Kishore, R. 2013. “Rules of Crowd- sourcing: Models, Issues, and Systems of Control,” Information Systems Management (30:1), pp. 2-20.

Schein, E. H. 1985. Career Anchors, San Diego, CA: University Associates.

Schenk, E., and Guittard, C. 2011. “Towards a Characterization of Crowdsourcing Practices,” Journal of Innovation Economics and Management (1), pp. 93-107.

Schmidt, F. A. 2013. “The Good, the Bad and the Ugly: Why Crowdsourcing Needs Ethics,” in Proceedings of the 3rd Inter- national Conference on Cloud and Green Computing, Los Alamitos: IEEE Computer Society, pp. 531-535.

Schultze, U., and Stabell, C. 2004. “Knowing What You Don’t Know? Discourses and Contradictions in Knowledge Manage- ment Research,” Journal of Management Studies (41:4), pp. 549-573.

Shaban, H. 2015. “Senator Warner Calls for New Policy to Address On Demand Economy,” BuzzFeed News, June 4, 2015 (http://www.buzzfeed.com/hamzashaban/on-demand-economy- collides-with-presidential-politics-in-war#.pdw2qGKn8N; accessed August 8, 2015).

Silberman, M., Irani, L., and Ross, J. 2010. “Ethics and Tactics of Professional Crowdwork,” XRDS: Crossroads (17:2), pp. 39-43.

Sivek, S. C. 2012. “Ladies’ Home Journal Ventures into Bold Crowdsourcing Experiment,” Mediashift, January 24 (http://www.pbs.org/mediashift/2012/01/ladies-home-journal- ventures-into-bold-crowdsourcing-experiment024.html).

Spreitzer, G. M. 1995. “Psychological Empowerment in the Work- place: Dimensions, Measurement, and Validation,” Academy of Management Journal (38:5), pp.1442-1465.

Steelman, Z. R., Hammer, B. I., and Limayem, M. 2014. “Data Collection in the Digital Age: Innovative Alternatives to Student Samples,” MIS Quarterly (38:2), pp. 355-379.

Stein, M. K., Galliers, R. D., and Whitley, E. A. 2014. “Twenty Years of the European Information Systems Academy at ECIS: Emergent Trends and Research Topics,” European Journal of Information Systems, forthcoming (http://www.palgrave-journals. c o m / e j i s / j o u r n a l / v a o p / n c u r r e n t / p d f / e j i s 2 0 1 4 2 5 a . p d f ; doi:10.1057/ejis,2015.25).

Thomas, K. W., and Velthouse, B. A. 1990. “Cognitive Elements of Empowerment: An ‘Interpretive’ Model of Intrinsic Task Motivation,” Academy of Management Review (15:4), pp. 666-681.

von Krogh, G., and Spaeth, S. 2007. “The Open Source Software Phenomenon: Characteristics That Promote Research,” The Journal of Strategic Information Systems (16:3), pp. 236-253.

Walldius, A., Sundblad, Y., and Borning, A. 2005. “A First Analysis of the Users Award Programme from a Value Sensitive Design Perspective,” in Proceedings of Critical Computing: Between Sense and Sensibility, The Fourth Decennial Aarhus Conference, Aarhus, Denmark, August.

Walsham, G. 1995. “The Emergence of Interpretivism in IS Research,” Information Systems Research (6:4), pp. 376-394.

Wastell, D., and White, S. 2010. “Facts, Myths and Thought-Styles ... and a Rallying Cry for Civic Engagement,” The Journal of Strategic Information Systems (19:4), pp. 307-318.

Whittington, R. 2014. “Information Systems Strategy and Strategy- as-Practice: A Joint agenda,” The Journal of Strategic Informa- tion Systems (23:1), pp. 87-91.

Wiener, N. 1950. The Human Use of Human Beings (2nd ed.), New York: Doubleday Anchor.

Wiener, N. 1954. The Human Use of Human Beings: Cybernetics and Society (rev. ed.), Boston: Houghton Mifflin.

Wiener, N. 1964. God & Golem, Inc.: A Comment on Certain Points Where Cybernetics Impinges on Religion, Cambridge, MA: MIT Press.

Williamson, V. 2014. “On the Ethics of Crowd-Sourced Research,” unpublished paper, Harvard University (http://scholar.harvard. edu/files/williamson/ files/mturk_ps_081014.pdf).

Yin, R. 1994. Case Study Research: Design and Methods (2nd ed.), Thousand Oaks, CA: Sage Publications.

About the Authors

Xuefei (Nancy) Deng is an associate professor of Information Systems at California State University, Dominguez Hills. Previously, she was an assistant professor of Information Tech- nology Management at the Shidler College of Business, University of Hawaii at Manoa. She received her Ph.D. in Information Systems from the Tepper School of Business at Carnegie Mellon University and M.B.A. from American University in Washington, DC. Nancy’s research interests include crowdsourcing, IT impact, IT workforce, knowledge management, and e-commerce. Her research has been published in Journal of Management Information Systems, Decision Support Systems, Information Systems Journal, Journal of Information Technology Case and Application Research, Journal of Information Systems, and International Journal of Project Management, among others. She cochairs the “Social Media and Enterprise” mini-track at the Hawaii International Conference on Information Sciences and served as a track cochair at the 2015 Decision Sciences Institute Annual Meeting. Nancy is an associate editor for Information and Organization and Journal of Organiza- tional Computing and Electronic Commerce, and serves on the editorial review board of Knowledge Management Research & Practice. Nancy served as the corresponding author for this paper.

K. D. Joshi is the Philip L. Kays Distinguished Professor of Information Systems at Washington State University. She received her Master of Science in Engineering from the University of Michigan and received her Doctor of Philosophy in Business Administration (Decision Sciences and Information Systems) from the University of Kentucky. K. D.’s research interests focus on IT workforce issues, knowledge management, crowdsourcing, IT- enabled innovation, value sensitive designs, and health IT. Her research is cited over 3,500 times according to Google Scholar. She has been a principle investigator or co-principle investigator on grants totaling over $5M from the National Science Foundation. Her research has appeared in journals such as Information Systems Research, Decision Support Systems, IEEE Transactions on Engi- neering Management, Communications of the ACM, Journal of Stra- tegic Information Systems, DATABASE, Information and Manage-

MIS Quarterly Vol. 40 No. 2/June 2016 301

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

ment, The Information Society, Journal of the American Society for Information Science and Technology, Information Systems Journal, IEEE Transactions on Professional Communication, Communica- tions of the AIS, and Journal of Organizational Computing and Electronic Commerce. She currently serves as a senior editor for Information Systems Journal and Information Systems Management, as a special section editor of Social Inclusion and IS for DATABASE, and as an associate editor for Communications of the AIS and Journal of Organizational Computing and Electronic Commerce.

Robert D. Galliers is Bentley University’s University Distinguished Professor having previously served as Provost. He also currently holds a fractional appointment as Professor of IS at Loughborough University. Previously, Bob has held professorships at the London School of Economics, Warwick Business School (where he served as Dean), and Curtin University. He has held a large number of

visiting professorships and serves on various university advisory boards internationally. His work has been published in many of the leading IS journals, including MIS Quarterly, Journal of the AIS, IEEE Transactions on Engineering Management, Communications of the ACM, Information Systems Journal, Journal of Information Technology, Information & Organization, European Journal of Information Systems, DATABASE, Journal of the American Society for Information Science & Technology, and Information & Manage- ment, and has been cited over 8,000 times according to Google Scholar. He is editor-in-chief of The Journal of Strategic Information Systems and is a Fellow of the British Computer Society, the Royal Society of Arts, and the Association for Information Systems, of which he was President in 1999. He received an Honorary Doctor of Science degree from Turku University, Finland in 1995 and the AIS LEO Award for exceptional lifetime achievement in 2012.

302 MIS Quarterly Vol. 40 No. 2/June 2016

SPECIAL ISSUE: ICT AND SOCIETAL CHALLENGES

THE DUALITY OF EMPOWERMENT AND MARGINALIZATION IN MICROTASK CROWDSOURCING: GIVING VOICE TO THE LESS

POWERFUL THROUGH VALUE SENSITIVE DESIGN

Xuefei (Nancy) Deng College of Business Administration and Public Policy, California State University, Dominguez Hills,

1000 E. Victoria Street, Carson, CA 90747 U.S.A. {ndeng@csudh.edu}

K. D. Joshi Carson College of Business, Washington State University, Todd Hall 42,

Pullman, WA 99164-4743 U.S.A. {joshi@wsu.edu}

Robert D. Galliers Information and Process Management/Sociology Departments, Bentley University, 174 Forest Street, Waltham, MA 02452 U.S.A. and

School of Business and Economics, Loughborough University, Loughborough, Leicestershire LE11 3TU U.K. {rgalliers@bentley.edu}

Appendix A

Crowdsourding Literature Review

Crowdsourcing (CS) emerges in a variety of forms (e.g., compensation-based, contest-based) and contexts (e.g., innovation, problem solving, and microtask). Not surprisingly, there exist a variety of definitions for CS, depending on the viewpoint adopted. One of the most frequently cited definitions is by Howe (2008, p. 1), who defines CS as “the act of taking a task traditionally performed by a designated agent (usually an employee or a contractor) and outsourcing it by making an open call to an undefined but generally large group of people.” Recent literature in CS has posited taxonomies for classifying diverse forms of CS (e.g., Estellés-Arolas and González-Ladrón-de-Guevara 2012; Osella 2014; Tarrell et al. 2013; Zhao and Zhu 2014). These studies use attributes such as the compensation received, the nature of the sourcing process, and the types of tasks in creating CS classifications. In this study, we view microtask CS as having followed three general characteristics (Estellés-Arolas and González-Ladrón-de-Guevara 2012): an undefined, generally large group of individuals (the crowd) who answer the open call made by job requesters (the initiator) and take on microtasks for micropayment (the process).

Research on microtask CS can be summarized around its four structures, namely governance, compensation, microtask, and technology. The governance structure refers to CS work practices, standards and policies. The compensation structure refers to payment arrangements. The microtask structure refers to the properties of CS jobs. The technology structure refers to the IT infrastructure used to build the CS work environment. Table A1 summarizes the relevant studies on microtask CS between 2010 and September 20141 under the four headings.

1While we included literature in our search from January 2006, most refereed conference and journal publications on the topic appeared from 2010 onward.

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A1

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table A1. Summary of Reviewed Studies on Microtask Crowdsourcing

Reference Publication Outlet Findings

CS Governance Structure

Greengard (2011) Communications of the ACM

CS is powerful because virtually anyone has the potential to plug in valuable information. CS offers businesses and developers access to an on-demand, scalable work force. CS continues to grow—and involves increasingly complex issues.

Schenk and Guittard (2011)

Journal of Innovation Economics & Management

Compares CS with established theories (open innovation, user inno- vation). Proposes and illustrates a typology of CS practices based on two criteria: the integrative or selective nature of the process and the type of tasks (routine, complex and creative) that are crowd sourced. Concludes that relying on the crowd is an appropriate method for organizations to use.

Kaganer et al. (2013) MIT Sloan Management Review

Companies face two major obstacles when sourcing work to the on- demand CS work force: perceived risk and limited capacity to handle large scale projects. Four models of human cloud platforms are proposed—aggregator; arbitrator; facilitator, and governor—based on two dimensions: who (buyer or platform) provides project gover- nance and where (the supplier or the platform) to place the buyers’ trust.

Vakharia and Lease (2013)

arXiv preprint arXiv:1310.1672.

Conducts cross-platform content analysis of seven crowd work platforms. Formulates key criteria for characterizing and differen- tiating crowd work platforms.

Saxton et al. (2013) Information Systems Management

Analyzes 103 well-known CS web sites. Develops a “taxonomic theory” of CS, resulting in nine distinct forms of CS model.

Nakatsu et al. (2014) Journal of Information Science

Follows an iterative approach and considers over 100 well-known examples of CS. Develops taxonomy of CS tasks with three dimensions of task complexity: (1) structure, (2) interdependence, and (3) commitment. Concludes with seven CS types.

Zogai et al. (2014) Journal of Business Economics

A case study of a German start-up CS intermediary that connects CS companies and the crowd. Challenges for CS intermediaries include managing the (1) process, (2) crowd, and (3) technology.

CS Compensation Structure

Finnerty et al. (2013) Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI, ACM.

CS is emerging as an effective method for performing tasks that require human skills (e.g., tagging photos, transcribing handwriting and categorizing data). Two experiments: (1) task design and reward scheme may affect work performance on CS platform and (2) guidelines for designing tasks with the aim to maximize worker performance.

Kittur et al. (2013) Proceedings of the 16th

ACM Conference on Computer Supported Cooperative Work

The crowd is a geographically distributed workforce to complete complex tasks on demand and at scale. Outlines a framework that will enable crowd work that is complex, collaborative and sustainable. Identifies research challenges in 12 major areas: workflow, task assignment, hierarchy, real-time response, synchronous collabora- tion, quality control, crowds guiding artificial intelligences (AIs), AIs guiding crowds, platforms, job design, reputation, and motivation.

CS Task Structure

Finnerty et al. (2013) Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI, ACM.

CS is emerging as an effective method for performing tasks that require human skills (e.g., tagging photos, transcribing handwriting and categorizing data). Two experiments: (1) task design and reward scheme may affect work performance on CS platform and (2) guidelines for designing tasks with the aim to maximize worker performance.

A2 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table A1. Summary of Reviewed Studies on Microtask Crowdsourcing (Continued)

Reference Publication Outlet Findings

Chandler et al. (2013) Handbook of Human Computation. Springer New York.

Investigates the role of microtask labor marketplaces in managing human and hybrid human machine computing. Distinct challenges in human computation (crowd): increased unsystematic error (e.g., mistakes) and systematic error (e.g., cognitive biases), both of which can be exacerbated when motivation is low, incentives are misaligned, and task requirements are poorly communicated.

Hassan and Curry (2013)

Proceedings of the 9th

International Conference on Collaborative Computing: Networking, Applications and Worksharing

Current CS task assignment primarily focuses on content-based approaches, qualifications, or work history. Proposes an alternative and complementary approach that focuses on capabilities workers employ to perform tasks to predict worker performance. Three different tasks: fact verification, image comparison and information extraction.

Sarasua and Thimm (2013)

Third International Conference on Cloud and Green Computing

Focuses on knowledge-intensive micro tasks in human computation scenarios. Profiles workers and proposes the introduction of a crowd worker CV as a comprehensive means to describe a worker’s expertise and interests.

CS Technology Structure

Kajino et al. (2014) Data Mining and Knowledge Discovery

Focuses on the privacy problems of CS workers. Proposes a protocol where a requester can estimate results while preserving worker privacy.

Geiger and Schader (2014)

Decision Support Systems Considers CS IS as sociotechnical systems that provide informa- tional products or services. Introduces personalized task recommen- dation mechanisms so that the CS task and workers’ individual interests and capabilities are better matched.

Saito et al. (2014) Universal Access in Human–Computer Interaction: Universal Access to Information and Knowledge (Volume 6, Part II)

Proposes a framework of microtasking that intrinsically supports the development of workers’ skills with three core modules: tutorial producer, task dispatcher, and feedback visualizer, all supported by a back-end skill assessment engine.

References

Chandler, J., Paolacci, G., and Mueller, P. 2013. “Risks and Rewards of Crowdsourcing Marketplaces,” in Handbook of Human Computation, New York: Springer, pp. 377-392.

Estellés-Arolas, E., and González-Ladrón-de-Guevara, F. 2012. “Towards an Integrated Crowdsourcing Definition,” Journal of Information Science (38:2), pp. 189-200.

Finnerty, A., Kucherbaev, P., Tranquillini, S., and Convertino, G. 2013. “Keep it Simple: Reward and Task Design in Crowdsourcing,” in Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI, New York: Association for Computing Machinery, Article 14.

Geiger, D., and Schader, M. 2014. “Personalized Task Recommendation in Crowdsourcing Information Systems—Current State of the Art,” Decision Support Systems (65), pp. 3-16.

Greengard, S. 2011. “Following the Crowd,” Communications of the ACM (54:2), pp. 20-22. Hassan, U., and Curry, E. 2013. “A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing,” in Proceedings

of the 9th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, October 20-23, Austin, TX, pp. 429-437.

Howe, J. 2008. “Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business,” The International Achievement Institute. Kaganer, E., Carmel, E., Hirschheim, R., and Olsen, T. 2013 “Managing the Human Cloud,” MIT Sloan Management Review (54:2), pp. 22-32. Kajino, H., Arai, H., and Kashima, H. 2014. “Preserving Worker Privacy in Crowdsourcing,” Data Mining and Knowledge Discovery (28:5-6),

pp. 1214-1335.

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A3

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Kittur, A., Nickerson, J. V., Bernstein, M. S., Gerber, E. M., Shaw, A., Zimmerman, J., Lease, M., and Horton, J. J. 2013. “The Future of Crowd Work,” in Proceedings of the 16th ACM Conference on Computer Supported Cooperative Work, January 2013, San Antonio, Texas.

Nakatsu, R. T., Grossman, E. B., and Lacovou, C. L. 2014. “A Taxonomy of Crowdsourcing Based on Task Complexity,” Journal of Information Science (40:6), pp. 823-834.

Osella, M. 2014. A Multi-Dimensional Approach for Framing Crowdsourcing Archetypes, unpublished Ph.D. Dissertation, Politecnico di Torino.

Saito, S., Watanabe, T., Kobayashi, M., and Takagi, H. 2014. “Skill Development Framework for Micro-Tasking,” in Universal Access in Human–Computer Interaction: Universal Access to Information and Knowledge (Volume 6, Part II), C. Stephanidis and M. Antona (eds.), New York: Springer International Publishing, pp. 400-409.

Sarusua, C., and Thimm, M. 2013. “Microtask Available, Send Us Your CV!,” in 2013 IEEE Third International Conference on Cloud and Green Computing, pp. 521-524.

Saxton, G. D., Oh, O., and Kishore, R. 2013. “Rules of Crowdsourcing: Models, Issues, and Systems of Control,” Information Systems Management (30:1), pp. 2-20.

Schenk, E., and Guittard, C. 2011. “Towards a Characterization of Crowdsourcing Practices,” Journal of Innovation Economics and Management (1), pp. 93-107.

Tarrell, A., Tahmasbi, N., Kocsis, D., Tripathi, A., Pedersen, J., Xiong, J., Oh, O., and de Vreede, G. J. 2013. “Crowdsourcing: A Snapshot of Published Research,” in Proceedings of the 19th Americas Conference on Information Systems, Chicago.

Vakharia, D., and Lease, M. 2013. “Beyond AMT: An Analysis of Crowd Work Platforms,” arXiv preprint (arXiv:1310.1672). Zhao, Y.,and Zhu, Q. 2014. “Evaluation on Crowdsourcing Research: Current Status and Future Direction,” Information Systems Frontiers

(16), pp. 417-434. Zogai, S., Bretschneider, U., and Leimeister, J. 2014. “Managing Crowdsourced Software Testing: A Case Study Based Insight on the

Challenges of a Crowdsourcing Intermediary,” Journal of Business Economics (84:3), pp. 375-405.

A4 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Appendix B

Research Methods

Survey Instrument

Survey on Crowdsourcing Work and Crowd Workers’ Experience

Dear Crowd Workers – Thank you for taking the time to complete the survey. We value your input and the important roles you play in helping us understand crowdsourcing work and crowd workers.

Are you a Mechanical Turk Master? G No G Yes

1. What types of HITs do you usually take? The following lists the seven categories of HITS on Mechanical Turk. Please indicate the extent to which you work on them (Scales: Quite a lot; Somewhat; Very few; Not at all).

(1) Data Processing HITs; (2) Categorization HITs; (3) Sentiment HITs; (4) Tagging HITs; (5) Content HITs; (6) Business Feedback HITs; (7) Academic Survey HITs.

2. What skills are required to perform the jobs you have identified above on Mechanical Turk? Please describe. 3. What do you like about doing crowdsourcing jobs on Mechanical Turk? And why? Please illustrate with examples. 4. What would you like to change about doing crowdsourcing jobs on Mechanical Turk? And why? Please provide examples. 8. Does doing crowd sourcing jobs on the Mechanical Turk allow you to attain the work values that are important to you in your career?

Why or why not? 9. What are your career goals? Describe the career pathway that is most attractive to you and most suitable to your work-life needs.

10. Does doing crowd sourcing jobs on Mechanical Turk allow you to meet your career goals? Why or why not? (If not, then please tell us why you are still working as a crowd worker.)

11. Overall, how satisfied are you with your crowdsourcing jobs on Mechanical Turk? Please explain. 12. What is your gender? G Male G Female 13. What is your age? 14. Which of the following best describes your highest achieved education level? (Select one)

( ) Some High School; ( ) High School Graduate; ( ) Some college, no degree ( ) Associates degree; ( ) Bachelors degree; ( ) Graduate degree (Master’s, Ph.D., etc.)

For your highest eduation degree identified above, what is your Field of Study?

15. What is the total income of your household? (Select one) ( ) Less than $25,000; ( ) 25,000 - $49,999; ( ) $50,000 - $74,999; ( ) $75,000 - $99,999; ( ) $100,000 or More

16. What is your current employment status? ( ) Employed Full-Time; ( ) Employed Part-Time; ( ) Un-employed; ( ) Other

17. How long (in months) have you been working as a crowd worker on Mechanical Turk? 18. On average, how many HITs do you usually take in a week on Mechanical Turk? 19. When do you usually schedule to work on the crowdsourcing jobs? And why? 20. On average, how many hours in a week do you spend on doing crowdsourcing work on Mechanical Turk? 21. Would you consider doing crowdsourcing work as your full-time job? Why or why not? 22. We welcome your feedback. Please use the space below if you have any additional comments or suggestions.

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A5

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table B1. Examples of Coding Discrepancies and Resolution

Examples Initial Coding and Resolution of Coding Discrepancies

“doing MTurk HITs indirectly helps in honing necessary skills I could later utilize to attain my goals”

During the initial coding, the two coders disagreed on their coding type: coder 1 considered it as “empowerment – competence” while coder 2 regarded it as “value – Access.” To resolve the issue, the two coders first discussed their rationale for the coding, and agreed the experience and feeling of empowerment is a product of presence of a value. In this case, the presence of the value “having access” led to the feeling of being empowered through gaining competence (e.g., “honing necessary skills”). Based on the discussion, the two coders reconciled the coding by agreeing that having access to opportunity is a necessary condition to empower the workers. They concluded that both the codes “access” value and “competence” empowerment should be included.

“I’m very satisfied with crowd sourcing jobs on MTurk. MTurk allows me to choose the jobs I want to complete. I only work on jobs that offer an excellent wage and allow me to work on a job that encourages me to think about the subject matter before responding to the questions asked.”

During the initial coding, the two coders both coded the statements “empowerment” but disagreed on their coding of the dimension of empowerment: coder 1 considered it “empowerment via self-determination” while coder 2 regarded it as “empowerment via meaning.” To resolve the coding on “empowerment via self-determination,” the coders reviewed the code definition and agreed that because the individual had a “choice,” there is thus empowerment via self-determination. To resolve the coding on “empowerment via meaning,” the two coders first discussed the code definition: “The job activities are personally meaningful to me” (e.g., important work for an individual; work that makes a person feel productive). However, the two coders read through the lines of the quote and decided that work choices enjoyed by a worker related directly to the nature of the work (e.g., “encourage me to think about the subject matter”) and compensation for the job (e.g., “offers an excellent wage). Then the two coders realized that there exist multiple dimensions (subcategories) of “meaningful,” viz.: economically meaningful, cognitively meaningful, in addition to the previous single dimension of meaningful. As a result, the two coders agree to add the two new sub- categories to the meaningful dimension under empowerment and modified the coding scheme accordingly.

A6 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table B2. Worker Values and Distribution by Demographic Factors

Panel A. Crowd Worker Values by Employment Status

Employed FT (n = 76)

Employed PT (n = 55)

Other (n = 27)

Unemployed (n = 52)

Total (n = 210)

Value Count % Count % Count % Count % Count %

Access 74 97% 53 96% 27 100% 50 96% 204 97%

Autonomy 70 92% 44 80% 22 81% 42 81% 178 85%

Fairness 45 59% 32 58% 15 56% 35 67% 127 60%

Transparency 25 33% 18 33% 5 19% 16 31% 64 30%

Communication 19 25% 14 25% 8 30% 14 27% 55 26%

Security 17 22% 10 18% 5 19% 11 21% 43 20%

Accountability 12 16% 10 18% 6 22% 14 27% 42 20%

Making an impact 13 17% 11 20% 3 11% 7 13% 34 16%

Dignity 10 13% 3 5% 2 7% 9 17% 24 11%

Panel B. Crowd Worker Values by Worker Gender

Female (n = 110) Male (n = 100) Total (n = 210)

Value Count % Count % Count %

Access 109 99% 95 95% 204 97%

Autonomy 91 83% 87 87% 178 85%

Fairness 72 65% 55 55% 127 60%

Transparency 30 27% 34 34% 64 30%

Communication 27 25% 28 28% 55 26%

Security 23 21% 20 20% 43 20%

Accountability 19 17% 23 23% 42 20%

Making an impact 19 17% 15 15% 34 16%

Dignity 12 11% 12 12% 24 11%

Panel C. Crowd Worker Values by Worker Education Level

High School (n = 27)

Some college (n = 62)

Associate degree (n = 25)

Bachelor’s degree (n = 73)

Graduate degree (n = 23)

Grand Total (n = 210)

Value Count % Count % Count % Count % Count % Count %

Access 27 100% 59 95% 25 100% 70 96% 23 100% 204 97%

Autonomy 26 96% 53 85% 20 80% 58 79% 21 91% 178 85%

Fairness 17 63% 39 63% 17 68% 45 62% 9 39% 127 60%

Transparency 4 15% 20 32% 8 32% 24 33% 8 35% 64 30%

Communication 1 4% 18 29% 3 12% 25 34% 8 35% 55 26%

Security 6 22% 11 18% 5 20% 14 19% 7 30% 43 20%

Accountability 2 7% 13 21% 3 12% 21 29% 3 13% 42 20%

Making an impact 4 15% 7 11% 6 24% 13 18% 4 17% 34 16%

Dignity 1 4% 7 11% 5 20% 9 12% 2 9% 24 11%

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A7

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table B2. Worker Values and Distribution by Demographic Factors (Continued)

Panel D. Crowd Worker Values by Worker Household Income

Less than $25,000 (n = 40)

$25,000– $49,999 (n = 82)

$50,000– $74,999 (n = 48)

$75,000– $99,999 (n = 23)

$100,000 or More (n = 17)

Total (n = 210)

Value Count % Count % Count % Count % Count % Count %

Access 37 93% 81 99% 47 98% 22 96% 17 100% 204 97%

Autonomy 32 80% 71 87% 44 92% 16 70% 15 88% 178 85%

Fairness 23 58% 51 62% 30 63% 15 65% 8 47% 127 60%

Transparency 12 30% 23 28% 21 44% 6 26% 2 12% 64 30%

Communication 12 30% 24 29% 13 27% 4 17% 2 12% 55 26%

Security 11 28% 11 13% 12 25% 6 26% 3 18% 43 20%

Accountability 10 25% 14 17% 13 27% 3 13% 2 12% 42 20%

Making an impact 7 18% 15 18% 4 8% 4 17% 4 24% 34 16%

Dignity 6 15% 10 12% 5 10% 3 13% 0 0% 24 11%

Panel E. Crowd Worker Values by Worker Age Group

Age 18–24

(n = 45) Age 25–30

(n = 51) Age 31–40

(n = 46) Age 41–50

(n = 45) Age 51+ (n = 23)

Total (n = 210)

Value Count % Count % Count % Count % Count % Count %

Access 42 93% 51 100% 46 100% 42 93% 23 100% 204 97%

Autonomy 38 84% 41 80% 40 87% 40 89% 20 87% 178 85%

Fairness 22 49% 36 71% 28 61% 28 62% 13 57% 127 60%

Transparency 13 29% 15 29% 13 28% 16 36% 7 30% 64 30%

Communication 10 22% 12 24% 12 26% 15 33% 6 26% 55 26%

Security 7 16% 12 24% 8 17% 13 29% 3 13% 43 20%

Accountability 8 18% 11 22% 13 28% 7 16% 3 13% 42 20%

Making an impact 3 7% 12 24% 8 17% 9 20% 2 9% 34 16%

Dignity 3 7% 6 12% 4 9% 9 20% 2 9% 24 11%

Appendix C

The Nine Crowd Worker Values

The crowd workers surveyed shared a common set of key values in the CS work environment. Our data revealed a set of nine values that they shared, namely, access, autonomy, fairness, transparency, communication, security, accountability, making an impact, and dignity. These values are associated with the work-related expectations of the crowd workers, as they interacted with, and engaged in, CS work on MTurk.

Access

Crowd workers in our sample valued access to various jobs opportunities available on MTurk; the opportunity to earn extra income was the most frequently cited benefit of microtask CS. Yet, the value of access is multifaceted; it conveys different meanings to different workers. It provides a means of making income for those who are unable to conform to traditional workplace expectations due to certain life circumstances (e.g., stay-at-home parents, individuals with health problems):

I’m not able to be hired for a job outside the home due to my health condition. I must work from home. If I am too sick to work [on MTurk] a certain day, I don’t have to worry about being fired for not working. If I can work extra [hours on

A8 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

MTurk] another day, I do it when the work is available. (Male, 49 years; Some college education; Household income < $25,000; Employment-Other: Disabled)

For some, CS was their only job option for making ends meet during periods of unemployment, creating an important, financial cushion for them and their families:

[CS] work is providing me money to pay monthly bills and helping me to dig out of the hole I was in from being out of work for so long. [MTurk] helps me to pay bills and buy groceries. It’s necessary until I get another job that pays more. (Female, 45 years; Bachelor’s degree; Household income $25,000–$49,999; Unemployed)

Autonomy

Crowd workers appreciated that MTurk provided them with flexibility and freedom in making job-related decision (e.g., deciding what tasks to take on, and how, where, and when to perform the tasks). While some enjoyed the total control over their work schedule on MTurk (e.g., scheduling HITs at their own pace instead of constrained by the 9 to 5 time frame), others appreciated the freedom in their choices of tasks. As one worker put it:

I enjoy being able to choose what I’ll work on. I have chosen assignments based purely on pay and profitability and… because the subject matter was interesting to me or for a good cause. (Female, 45 years; Bachelor’s degree; Household income $25,000–$49,999; Employed full-time)

Fairness

Crowd workers also valued fairness in being compensated and treated in conducting microtask CS. Our data analysis revealed a perception of a lack of fairness in microtask compensation, however. While most of the respondents appreciated the fact that they could obtain monetary compensation (“The best thing about doing [CS] jobs on [MTurk] is getting paid”) the same individuals felt they were unfairly compensated and sometimes unfairly treated by job requesters.

I think all requesters should put HITs at a minimum of [a] $6/hr rate or more. For example, a survey that takes 10 minutes but only pays 20 cents is ridiculous! (Male, 18 years; Some college education; Household income $75,000–$99,999; Unemployed)

In addition to the unfair pay rates, some crowd workers also perceived unfairness in MTurk’s evaluation of workers (i.e., MTurk’s Master Qualification program). Moreover, some crowd workers felt that they were being unfairly treated by job requesters who rejected their work without clear reasons.

Transparency

Open and transparent work standards and protocols in microtask CS are appreciated by the crowd workers in our sample. To some extent, the value of transparency is embedded in the open sourcing form of work because each HIT published on MTurk includes a brief job description, instructions, time requirements, and payment amount. However, sometimes, workers felt they were blind to the process and their job performance, for example:

I would change how requesters are reviewed and rated so that Turkers like myself can avoid bad requesters and do quality work for the ones that are worth it. For example, as of now, we can’t see ratings of [requesters] on the site, and last month I was rejected unfairly by a requester [who] just wanted to keep the work and not pay…so we’re “blind’”and can’t know that a requester can potentially reject our content. Having a rating platform (like Turkopticon) benefits everyone working for the site. (Male, 25 years; Bachelor’s degree, Household income $25,000–$49,999; Employed full-time)

Because this lack of transparency was related to MTurk’s management of the CS process, there was a call for MTurk to establish governance structures to enforce standards that are clear and understandable by both workers and job requesters. For example, to improve transparency and job feedback in the CS marketplace, some crowd workers urged MTurk to establish a requester screening/evaluation system. Because of the lack of a systematic reputation mechanism for requesters on MTurk, workers relied on off-site, third party reputation systems. Turkopticon (www.turkopticon.com) mentioned above is one such site that allows workers to rate job requesters on four aspects (communicativity,

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A9

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

generosity, fairness, and promptness), thus enabling workers to learn about the reputations of job requesters before deciding to work for them. Building such a reputation system on the MTurk site itself would be greatly appreciated by the workers.

Communication

Crowd workers expressed their desire for direct and open communication with job requesters so as to be informed about their job outcomes and to reduce the risk of potential disputes. Their desire to receive job feedback grew stronger when their payments were rejected without reason because each rejection received negatively impacts efforts to attain Master status. As one worker urged:

I’d like to see better communication between requesters and workers. Amazon doesn’t get involved in disputes or misunderstandings between workers and requesters. For the most part, the requesters who I have needed to contact have been polite and have made a point to respond to me, which I appreciate. (Female, 28 years; Some college education; Household income $50,000–$74,999; Unemployed)

However, most workers found the existing communication tools on MTurk insufficient because the tools did not facilitate their interaction with job requesters in order to clarify a HIT or check a job outcome. In such cases, crowd workers worked around MTurk’s communication tools by directly e-mailing a job requester to clarify a HIT and to avoid a potential rejection, as a stay-at-home mom has done in her urgent email to a job requester:

My daughter just hit the enter key on me early! I was in the middle of answering question number [xx]. Please! Please! Please! Is there a way I can get back into the survey?? I really need to complete it and earn the money. (Female, 23 years; Associate degree; Household income $25,000–$49,999; Employment-Other: Stay-at-home Mom)

Security

A CS work environment that provides assurance, safety, and reduces work disruptions is highly desirable. Some crowd workers perceived the CS job on MTurk a secure one as they perceived MTurk less likely of “going bankrupt.” Nevertheless, lack of job security is one of the most prevalent feelings shared by crowd workers. This is often due to task scamming, which causes disruption in pay and a potential threat in undermining the crowd workers’ efforts and reputation:

I would like MTurk to be more protective of workers—we get scammed a lot. Say you spend 30 minutes filling out a survey, but after you submit your answers, you get no completion code to get paid. They have their data and you get nothing. (Female, 43 years; High school graduate; Household income $75,000–$99,999; Employment-Other: Stay-at- home mom)

Accountability

Crowd workers hope that the actions of people or institutions be traced uniquely to individual workers and job requesters so that crowd workers and job requesters be held responsible for their work and behavior. As indicated by some respondents, the current design of the MTurk platform rarely held job requesters accountable for their behavior; for example:

The pay should be adjusted to at least minimum wage levels for all jobs because often requesters lie about pay in order to get you invested in their HITs. Requesters should be held accountable for their shortcomings/unethical behavior because too often they abuse a system that does not care. (Male, 24 years; Bachelor’s degree; Household income $75,000– $99,999; Employed part-time)

Making an Impact

The desire to contribute to the community and to have a positive impact on other people’s lives is shared by crowd workers. In such a large online labor marketplace, as is the case with CS, individual workers become part of a community where they have opportunities to influence others via their work. The fulfillment of this social need is enabled by the MTurk platform. In particular, the value of making an impact can be supported by performing certain types of micro tasks, such as research-related, survey HITs. As one worker explained:

A10 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

[Working on academic surveys] gives me a sense of pride knowing that I’m helping the research community by assisting them with data collection. For example, I really enjoy surveys that are noted as being for Master’s or Doctoral research, because I know that someone is working to not only provide new, insightful research to the world, but to also better themselves. (Male, 30 years; Bachelor’s degree; Household income $75,000–$99,999; Employed part-time)

Dignity

Crowd workers value a sense of pride in self. This desire for dignity (pride and respect) is illustrated in the following two remarks. Whereas the first worker felt that his work was valued and respected by his job requester, the second worker held a different view:

Some requesters actually value our input highly, and they compensate accordingly. Some actually “get it” that there are many of us who are doing this to keep the lights on. (Male, 49 years; Some college education; Household income < $25,000; Unemployed)

I don’t feel like there is enough respect for workers. For example, someone might offer $0.50 for an hour’s worth of work. Requesters can often be ignorant of the ins and outs of MTurk and this can lead to unwarranted rejects. (Male, 34 years; Bachelor’s degree; Household income $50,000–$74,999; Employed full-time)

In sum, the CS platform on MTurk mostly fulfills workers’ values in relation to access, autonomy, and making an impact. This is so because the platform offers people from all walks of life free access to micro tasks and provides them with control over their CS job decisions. The remaining value categories were found to be partially supported, or in some cases, rarely supported.

Appendix D

The Four Cognitions of Empowerment

Table D1. The Four Cognitions of Empowerment Experienced by Crowd Workers

Employment Status

Empowerment

Employed Full- time

(n = 76)

Employed Part-time (n = 55)

Employment Other

(n = 27) Unemployed

(n = 52) Total

(n = 210)

Meaning 97% 96% 96% 94% 96%

Self-Determination 93% 80% 81% 81% 85%

Impact 16% 20% 11% 13% 16%

Competence 14% 15% 22% 12% 15%

Educational Level Attained

Empowerment

High School Graduate (n = 26)

Some College

Education (n = 62)

Associate Degree (n = 25)

Bachelor’s Degree (n = 73)

Graduate Degree

(Master’s; Doctorate)

(n = 23) Total

(n = 210)

Meaning 100% 95% 100% 96% 96% 96%

Self-Determination 100% 87% 80% 79% 91% 85%

Impact 15% 11% 24% 16% 17% 16%

Competence 19% 11% 4% 18% 22% 15%

Note: Percentages relate to the proportion of the total number of workers (listed as n in the column heading) in each category.

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A11

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table D2. Associations Between Worker Values and Forms of Empowerment

Value\Empowerment Self-

Determination Meaning Impact Competence

Total Workers Who Expressed

the Value

Access 166 (81.4%) 202 (99.3%) 44 (21.4%) 34 (16.6%) 204

Autonomy 178 (100%) 117 (65.7%) 21 (11.8%) 25 (14.0%) 178

Fairness 109 (85.8%) 91 (71.7%) 25 (19.7%) 23 (18.1%) 127

Transparency 56 (87.5%) 47 (73.4%) 10 (15.6%) 7 (10.9%) 64

Communication 49 89.1%) 35 (63.6%) 8 (14.5%) 6 (10.9%) 55

Security 37 (86.0%) 31 (72.1%) 10 (23.3%) 12 (27.9%) 43

Accountability 34 (81.0%) 31 (73.8%) 8 (19.0%) 8 (19.0%) 42

Making an Impact 22 (64.7%) 32 (94.1%) 33 (97.1%) 7 (20.6%) 34

Dignity 18 (75.0%) 17 (70.8%) 2 (8.3%) 1 (4.2%) 24

Note: The % in the brackets are the percentage of the row total; it represents the percentage of all workers sharing the same value that

experienced a certain form of empowerment.

Appendix E

The Four Types of Marginalization

Table E1. The Four Types of Marginalization Experienced by Crowd Workers

Employment Status

Marginalization

Employed Full-time (n = 76)

Employed Part-time (n = 55)

Employment Other

(n = 27) Unemployed

(n = 52) Total

(n = 210)

(1) Economic 59% 58% 56% 67% 60%

(2) Institutional policies and practices 33% 40% 37% 42% 38%

(3) Institutional technical features 16% 22% 37% 27% 23%

(4) Competence 12% 4% 27% 10% 9%

Educational Level Attained

Marginalization

High School Graduate (n = 26)

Some College

Education (n = 62)

Associate Degree (n = 25)

Bachelor’s Degree (n = 73)

Graduate Degree

(Master’s, Ph.D.)

(n = 23) Total

(n = 210)

(1) Economic 65% 63% 68% 62% 39% 60%

(2) Institutional policies and practices 31% 42% 36% 34% 48% 38%

(3) Institutional technical features 31% 23% 20% 21% 26% 23%

(4) Competence 8% 10% 8% 4% 22% 9%

Note: Percentages relate to the proportion of the total number of workers (listed as n in the column heading) in each category.

A12 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table E2. Associations Between Worker Values and Forms of Marginalization

Value/Marginalization Economical Institutional Governance

Technical Artifact Competence

Total Workers Who Expressed

the Value

Access 103 (50.3%) 79 (38.6%) 52 (25.5%) 14 (6.9%) 204

Autonomy 83 (46.1%) 69 (38.8%) 39 (21.9%) 17 (9.6%) 178

Fairness 98 (77.2%) 58 (45.7%) 36 (28.3%) 7 (5.5%) 127

Transparency 27 (42.2%) 49 (76.6%) 26 (40.6%) 6 (9.4%) 64

Communication 20 (36.4%) 41 (74.5%) 20 (36.4%) 7 (10.9%) 55

Security 23 (53.5%) 31 (72.1%) 20 (46.5%) 7 (16.3%) 43

Accountability 22 (52.4%) 40 (95.2%) 21 (50.0%) 4 (9.5%) 42

Making an Impact 21 (61.8%) 15 (44.1%) 12 (35.3%) 1 (2.9%) 34

Dignity 17 (70.8%) 10 (41.7%) 4 (16.7%) 2 (8.3%) 24

Note: The percentages in the brackets are the percentage of the row total; it represents the percentage of all workers sharing the same value that experienced a certain form of marginalization.

Appendix F

Distribution of Crowd Workers Experiencing the Duality of Empowerment and Marginalization

Those CS workers who attained a sense of self-fulfillment from their work tended to feel empowered, while those experiencing unfair compensation had a strong sense of economic marginalization. Among all of the 210 respondents, 58 percent of them simultaneously felt empowered because of doing meaningful CS work and felt that the rates of pay were low (economic marginalization), as shown in Table F1.

Table F1. Duality: Four Types of Empowerment Versus Four Types of Marginalization

Marginalization

Empowerment Economic Institutional

Technical Features Institutional Policies

and Practices Competence Total

Meaning 121 (58%) 46 (22%) 75 (36%) 14 (7%) 202 (96%)

Self-Determination 109 (52%) 39 (19%) 69 (33%) 17 (8%) 179 (85%)

Impact 25 (12%) 12 (6%) 15 (7%) 1 (0.5%) 33 (16%)

Competence 23 (11%) 14 (7%) 15 (7%) 3 (1%) 31 (15%)

Total 127 (60%) 48 (23%) 79 (38%) 18 (9%) 210 (100%)

Note: The percentages in the brackets are the percentage of the total 210 respondents.

The duality experienced by the majority of the crowd workers we surveyed is distributed relatively evenly across the demographic factors we measured. For the most part, duality is experienced reasonably uniformly between workers of different gender (Women: 72%; Men: 68%); among workers with varying levels of education (High school graduate: 77%; Some college education: 77%; Associate degree: 80%; Bachelor’s degree: 66%; graduate degree: 48%); workers with different employment statuses (Full-time: 71%; Part-time: 65%; Other: 63%; Unemployed: 77%); workers from different age groups (Aged 18–24: 62%; 25–30: 84%; 31–40: 65%; 41–50: 69%; 51+: 65%), and workers with diverse household income levels (< $25,000: 75%; $25,000–$49,999: 72%; $50,000–$74,999: 73%; $75,000–$99,999: 70%; $100,000+: 41%).

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A13

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table F2. Distribution of Crowd Workers with Duality (n = 147)

Gender Female Male Total

Employees w/duality 79 68 147

Total Employees 110 100 210

% of Duality Cases 72% 68% 70%

Employment Status Employed Full-Time

Employed Part-Time Other Unemployed Total

Employees w/duality 54 36 17 40 147

Total Employees 76 55 27 52 210

% of Duality Cases 71% 65% 63% 77% 70%

Education Level High School

Graduate Some

college Associate

degree Bachelor’s

degree Graduate degree (Master’s,Ph.D.) Total

Employees w/duality 20 48 20 48 11 147

Total Employees 26 62 25 73 23 210

% of Duality Cases 77% 77% 80% 66% 48% 70%

Age Group Age 18–24 Age 25–30 Age 31–40 Age 41–50 Age 51+ Total

Employees w/duality 28 43 30 31 15 147

Total Employees 45 51 46 45 23 210

% of Duality Cases 62% 84% 65% 69% 65% 70%

Household Income Less than $25,000

$25,000– $49,999

$50,000– $74,999

$75,000– $99,999 $100,000 or More Total

Employees w/duality 30 59 35 16 7 147

Total Employees 40 82 48 23 17 210

% of Duality Cases 75% 72% 73% 70% 41% 70%

We also examined those respondents who only expressed feelings of empowerment and found that those cases were almost uniformly distributed across each of the demographic categories with the exception of household income greater than $100,000 and education background with graduate degree. This is reflected in Table F3.

A14 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table F3. Distribution of Crowd Workers with Empowerment Only (n = 61)

Gender Female Male

Empowerment Only 30 31 61

Total Respondents 110 100 210

% of Empower-Only 27% 31% 29%

Employment Status Employed Full-Time

Employed Part-Time Other Unemployed Total

Empowerment Only 22 18 10 11 61

Total Respondents 76 55 27 52 210

% of Empower-Only 29% 33% 37% 21% 29%

Education Level High School

Graduate Some college

education Associate

degree Bachelor’s

degree Graduate degree (Master’s, Ph.D.) Total

Empowerment Only 7 14 5 24 11 61

Total Respondents 26 62 25 73 23 210

% of Empower-Only 27% 23% 20% 33% 48% 29%

Age Group Age 18–24 Age 25–30 Age 31–40 Age 41–50 Age 51+ Total

Empowerment Only 16 8 15 14 8 61

Total Respondents 45 51 46 45 23 210

% of Empower-Only 36% 16% 33% 31% 35% 29%

Household Income Less than $25,000

$25,000– $49,999

$50,000– $74,999

$75,000– $99,999

$100,000 or More Total

Empowerment Only 10 23 13 6 9 61

Total Respondents 40 82 48 23 17 210

% of Empower-Only 25% 28% 27% 26% 53% 29%

Appendix G

Ethical Considerations and the Design of Microtask CS: A Research Agenda

Gregor and Hevner (2013), in their recent paper, state that useful knowledge of design science research can be divided into two distinct types: descriptive and prescriptive knowledge. The descriptive is the what knowledge about phenomena, whereas prescriptive knowledge is the how knowledge of building artifacts. They argue that both types of knowledge are required to build a comprehensive knowledge base for a particular design science research domain. We contribute to design science research by extending the descriptive body of knowledge about the ethical design and use of microtask CS. This knowledge can be applied in the conduct of future studies that can help develop prescriptive knowledge about building new (and/or refining the existing) microtask CS artifacts.

In addition, the worker values and CS platform structures uncovered in this study lay the groundwork for future research. Specifically, this work could help advance ethical considerations in the design of microtask CS platforms. We propose two kinds of analysis that can be conducted to further this work, based on the practical suggestions provided in the VSD literature (Friedman et al. 2008). First, future research can conduct value-driven investigations on microtask crowdsourcing platforms where a more detailed and nuanced tripartite analysis is done on each value. Second, a similar, structure-driven analysis can be undertaken by focusing on one of the four structures where the values are implicated. We should point out that VSD also suggests a third kind of analysis, which is tool-driven, where the focus is on investigating a tool using the three-part analysis (such as a software application for batch processing of micro tasks on MTurk). However, we believe that such an analysis overlaps with the technical investigations of value- and/or structure-driven analysis. In addition, VSD instructs the use of direct stakeholders (in our case, crowd workers, requestors, MTurk) and indirect stakeholders (people whose jobs are being crowd sourced or who are excluded from this environment because of the digital divide) while conducting design investigations. Given our study’s focus on crowd workers’ views, we propose future research that is based on this perspective. The aforementioned two research approaches, together with some potential broad research questions, are discussed below.

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A15

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Value-Driven Analysis: The process of value-driven analysis through a series of iterative studies is well-articulated in the VSD literature. For instance, Friedman’s studies on informed consent in online transactions (Friedman, Felten, Millett 2000; Friedman, Howe, and Felten 2002; Friedman, Kahn, and Howe 2000; Friedman and Millett 1995; Millett et al. 2001) provide good guidance on how research through value driven investigations can unfold. Table G1 presents some of the key macro-level research problems, providing a template for future work. The conceptual value-driven investigations entail a review of relevant literature. In particular, “the philosophical ontological literature can help provide criteria for what a value is, and thereby how to assess it empirically” (Freidman et al. 2008, p. 89). The empirical investigations test the human response to value conceptualizations by eliciting the perspectives of the key stakeholders using methodologies prescribed by the social sciences. The technical investigations assess how a value is implicated (through retrospective analysis) or can be implicated (through prospective designs) in technological properties and underlying mechanisms.

Structure-Driven Analysis: As illustrated in Table G2, the structure-driven investigations can begin at one of the four structures (microtask, governance, compensation, technology) where the values are embodied. For each of these structures, a conceptual, empirical, and technical analysis can be conducted to propose designs that empower workers with minimum marginalization. While value-driven analysis is well articulated in the VSD literature, the guidance on structure-driven analysis is less clear. Here, we detail how future research can build on the empirical investigation conducted in this study.

Table G1. Future Potential Value-Driven Investigations of Microtask Crowdsourcing Platforms

Values Conceptual Investigations Empirical Investigations Technical Investigations

Unit of Analysis: Value Concept Objective: Entails philosophical and theoretical investigation of a value through the lenses of ethical philosophies and past theories/ literature relevant to microtask CS.

Unit of Analysis: Humans Objective: Entails examining the human response to a value in the context of microtask CS platforms through social science research method- ologies such as surveys, interviews, observations.

Unit of Analysis: Technology Objective: Entails retrospec- tive analysis of current micro- task CS technologies to assess the nature of value implications and/or building new prospective designs to implicate a value in these platforms.

Access Provide more detailed and nuanced working conceptualiza- tions derived from the literature for each of the values revealed by the crowd workers. Clarify fundamental issues under- pinning in the values revealed by the crowd workers in a manner that provides a basis for com- paring results across research studies.

Provide guidelines to resolve value conflicts (empowerment and marginalization) to arrive at trade- offs among competing values.

Examine the comprehensiveness and completeness of these derived values. Are there additional values that were not revealed in this research?

Study how workers prioritize competing values while engaging in crowd work.

Examine how existing techno- logical properties and under- lying mechanisms and tools support or hinder a human value.

Conduct retrospective analyses of current tools and technologies to examine the extent to which the values uncovered in this work are implicated in various microtask CS platforms.

Proactively design proof of concept for systems to support values identified in our investigation.

Accountability

Autonomy

Communication

Dignity

Fairness

Making an impact

Security

Transparency

A16 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Table G2. Future Potential Structure-Driven Investigations of Microtask Crowdsourcing Platforms

Values Conceptual Investigations Empirical Investigations Technical Investigations

Unit of Analysis: Structure Objective: Entails philosophical and theoretical investigation of values through the lenses of ethical philosophies and past theories/literature relevant to a particular structure.

Unit of Analysis: Humans Objective: Entails examining the human response to values embodied in the structure through social science research methodologies such as surveys, interviews, observations.

Unit of Analysis: Technology Objective: Entails examining how existing technological properties & underlying mechanisms used to build a certain structure support or hinder human values.

Microtask Structure

Use work/job design, job crafting; motivation, batch design and pro- cessing theories to conceptualize design components for this struc- ture that help in realizing the values important to the crowd workers.

Examine how the posited conceptual microstask designs, once implicated in supporting technologies, are received by the crowd workers.

Proposes new tools and technologies to apprehend crowd worker values relevant to the microtask structure.

Governance Structure

Use procedural justice theories, agency theories; dispute resolu- tion literature to conceptualize design components for this structure that help in realizing the values important to the crowd workers.

Examine how the posited conceptual designs for gover- nace, once implicated in supporting technologies, are received by the crowd workers.

Proposes new tools and technologies to apprehend crowd worker values relevant to the governance structure.

Compensation Structure

Use distributive justice theories, equity theories to conceptualize design components for this structure that help in realizing the values important to the crowd workers.

Examine how the posited conceptual designs for compensation, once implicated in supporting technologies, are received by the crowd workers.

Proposes new tools and technologies to apprehend crowd worker values relevant to the compensation structure.

Technology Structure

Use work pattern literature, such as preference theory, boundary- less careers, computer-mediated work literature to conceptualize platform designs that could adapt to a variety of work pattern preferences.

Examine how the posited conceptual designs for technology, once implicated in supporting technologies, are received by the crowd workers.

Conduct value suitability analysis to evaluate the limits of IT in empowering and managing marginalization; identifying those values that can realistically be implicated in the technology.

Our empirical assessment suggests that the microtask structure provides work pattern flexibility by affording workers with the freedom to choose where, how and when they work, whereas task design focuses on what they work on. Our results also show that crowd workers’ task preferences differ. Some look for tasks that meet their prosocial needs, which make them feel that they are having a positive impact on society, whereas others look for tasks that fulfill their growth needs (skill variety); for example, while they are in midst of career transitions (through job loss, career changes, elderly or child care responsibilities, etc.) or desire to qualify for MTurk’s Master program. Future work needs to focus on conceptual investigations of this structure by leaning on current theoretical perspectives in this area, such as job enrichment (Oldman and Hackman 2010); job crafting (Berg et al. 2010); batch design and processing (Laguna and Marklund 2013). The job enrichment literature shows that certain task characteristics, such as task variety, task identity, and task significance, can help empower workers by fulfilling their values (Hackman and Oldham 1975, 1976). Therefore, future technical investigations could build on job enrichment theories by generating ways of crafting CS tasks (Wrzesniewski and Dutton 2001) that empower workers and prevent requesters from following a Taylorist-style doctrine (Taylor 1911), slicing and dicing jobs into micro tasks with the sole goal of gaining efficiencies and control.

The findings arising from this study illustrate that crowd worker responses to MTurk’s governance structure were less favorable and thus account for a considerable portion of feelings of marginalization. While crowd workers appreciate control over their work patterns and the nature of tasks they perform, they are less appreciative of the lack of sensitivity toward fair and transparent governance processes and polices

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A17

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

embodied in the CS platform. This displeasure is often a result of not having an avenue to voice complaints and resolve disputes with the requesters. The crowd workers who responded to our survey expressed the need for design features in the CS platform that would allow them to protect themselves from requesters that have a bad reputation (task scams, missed payments, low pay). In addition, they also desired on the CS platform digitized mechanisms that would not only allow them to communicate directly and openly with the requesters, but also support fair dispute resolution. Future design investigations should be sensitive to the issues that help reduce uncertainty and provide greater predictability as to how crowd workers’ performance is evaluated. Future conceptual investigations could derive conceptualizations from justice theory (Rawls 1971) and related organizational justice theories (Gilliland et al. 2001) that focus on procedural justice to inform prospective future technical investigations whose aim is to design governance structures for microtask CS platforms that mitigate the marginalization of crowd workers. Future work could also conduct retrospective technical investigations of tools and applications currently built into microtask CS platforms.

Similar to the governance structure, the compensation structure was salient to crowd workers’ perceptions of marginalization. In addition to fair and transparent processes, crowd workers are also sensitive about equitable, transactional practices that ensure just allocation of compensation for task completion. Our respondents voiced concerns regarding payments and compensation that did not dignify their work efforts. Many tasks did not even meet the minimum wage rate. Moreover, there were no payment guarantees upon task completion in certain instances. Future investigations could help conceptualize compensation and payment procedures and regulations that ensure a fair distribution of wealth generated within the crowdsourcing markets that would collectively benefit the workers, the requesters, and the platform owners. These conceptualizations can then be used to conduct future empirical and technical investigations. In the absence of such procedural and regulatory design options, these marketplaces will likely provision a race to the bottom where governments, in the name of a free market economy, could well cut back on regulation and enforcement of decent working conditions in order to lower labor costs. Future conceptual investigations could derive design specifications from equity theory (Adams 1965) and related organizational justice theories (Cowherd and Levine 1992) that focus on distributive justice, and not just focus research on common labor economic issues of reservation wage; in other words, the smallest wage a worker is willing to accept for a task (e.g., Horton and Chilton 2010).

Unlike governance and compensation structures, the technology structure was viewed favorably by the crowd workers in our study. One of the principle features germane to this kind of CS work environment is the free and open access to online jobs. This technological feature broadens labor force participation by allowing nontraditional labor (such as stay-at-home parents, the unemployed, the physically impaired, retirees) to remain actively engaged in the labor market. This nontraditional labor force is turned away from traditional labor markets not because they are unwilling or unable to work, but rather because traditional forms of work are incongruent, and therefore in conflict, with their life circumstances or choices (Muffels 2005; Muffels and Luijkx 2005). Although the current CS platform mostly empowers crowd workers by fulfilling the values of autonomy and access, there is still room for improvement. It seems there is a greater amount of flexibility in workers choosing where they work (at home, in transit, in a café), and how they work (no dress code, how they sequence their tasks), but there are still constraints on when they work. For instance, one stay-at-home parent wished to have greater flexibility in task scheduling, noting that the short times allotted on HITs prevented her from accepting those tasks that she would like to conduct because they conflict with her child care responsibilities. Thus, conceptual investigations could help in formulating empirical investigations and testing technical designs that further adapt to work patterns congruent with this diverse workforce. Future design research could achieve this by building on the work pattern literature, such as preference theory (Hakim 2000) and boundaryless careers (Arthur and Rousseau 1996) to develop such conceptualizations. This structure would also benefit from literature on computer-mediated work, which could help in deriving value suitability analysis of CS platforms to recognize what value can be implicated in the technology and, more importantly, what cannot be implicated as this would limit the platform’s ability to empower and manage marginalization.

References

Adams, J. S. 1965. “Inequity in Social Exchange,” in Advances in Experimental Psychology, L. Berkowitz (ed.), New York: Academic Press, pp. 267-299.

Arthur, M. B., and Rousseau, D. M. (eds.). 1996. The Boundaryless Career: A New Employment Principle for a New Organizational Era, Oxford, UK: Oxford University Press.

Berg, J. M., Wrzesniewski, A., and Dutton, J. E. 2010. “Perceiving and Responding to Challenges in Job Crafting at Different Ranks: When Proactivity Requires Adaptivity,” Journal of Organizational Behavior (31:2-3), pp. 158-186.

Cowherd, D. M., and Levine, D. I. 1992. “Product Quality and Pay Equity Between Lower-Level Employees and Top Management: An Investigation of Distributive Justice Theory,” Administrative Science Quarterly (37:2), pp. 302-320.

Friedman, B., Felten, E., and Millett, L. I. 2000. “Informed Consent Online: A Conceptual Model and Design Principles,” CSE Technical Report 2000-12-2, University of Washington.

Friedman, B., Howe, D. C., and Felten, E. 2002. “Informed Consent in the Mozilla Browser: Implementing Value-Sensitive Design,” in Proceedings of the 35th Annual Hawaii International Conference on System Science, Los Alamitos, CA: IEEE Computer Society.

A18 MIS Quarterly Vol. 40 No. 2—Appendices/June 2016

Deng et al./Empowerment and Marginalization in Microtask Crowdsourcing

Friedman, B., Kahn Jr., P. H., and Borning, A. 2008. “Value Sensitive Design and Information Systems,” in The Handbook of Information and Computer Ethics, K. Himma and H. Tavani (eds.), Hoboken, NJ: Wiley, pp. 69-102.

Friedman, B., Kahn, Jr., P. H., and Howe, D. C. 2000. “Trust Online,” Communications of the ACM (43:12), pp. 34-40. Friedman, B., and Millett, L. 1995. “It’s the Computer’s Fault—Reasoning About Computers as Moral Agents,” in Conference Companion

of the Conference on Human Factors in Computing Systems, New York: Association for Computing Machinery Press, pp. 226-227. Gilliland, S., Steiner, D. D., and Skarlicki, D. (eds.). 2001. Theoretical and Cultural Perspectives on Organizational Justice, Charlotte, NC:

Information Age Publishing. Gregor, S., and Hevner, A. R. 2013. “Positioning and Presenting Design Science Research for Maximum Impact,” MIS Quarterly (37:2), pp.

337-356. Hackman, J. R., and Oldham, G. R. 1975. “Development of the Job Diagnostic Survey,” Journal of Applied Psychology (60:2), pp. 159-170. Hackman, J. R., and Oldham, G. R. 1976. “Motivation Through the Design of Work: Test of a Theory,” Organizational Behavior and Human

Performance (16:2), pp. 250-279. Hakim, C. 2000. Work-Lifestyle Choices in the 21st Century: Preference Theory, Oxford, UK: Oxford University Press. Horton, J. J., and Chilton, L. B. 2010. “The Labor Economics of Paid Crowdsourcing,” in Proceedings of the 11th ACM Conference on

Electronic Commerce, Boston, June 7-11, pp. 209-218. Laguna, M., and Marklund, J. 2013. Business Process Modeling: Simulation and Design, Boca Raton, FL: CRC Press. Millett, L. I., Friedman, B., and Felten, E. 2001. “Cookies and Web Browser Design: Toward Realizing Informed Consent Online,” in

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York: ACM. Muffels, R. 2005. “Labour Market Mobility and Empowerment Patterns,” Position Paper Workpackage 3: Researching Labour Market

Transitions in European Welfare Regimes, Amsterdam: SISWO/Social Policy Research. Muffels, R. J. A., and Luijkx, R. 2005. “Job Mobility and Employment Patterns across European Welfare States: Is There a ‘Trade-Off’ or

a ‘Double Bind” Between Flexibility and Security?,” TLM.NET 2005 Working Paper No. 2005-13, SISWO/Social Policy Research, Amsterdam.

Oldham, G. R., and Hackman, J. R. 2010. “Not What it Was and Not What it Will Be: The Future of Jo Design Research,” Journal of Organizational Behavior (31:2-3), pp. 453-479.

Rawls, J. 1971. A Theory of Justice, Cambridge, MA: The Belknap Press. Taylor, F. W. 1911. Principles of Scientific Management, New York: Harper Wrzesniewski, A., and Dutton, J. E. 2001. “Crafting a Job: Revisioning Employees as Active Crafters of Their Work,” The Academy of

Management Review (26:2), pp. 179-201.

MIS Quarterly Vol. 40 No. 2—Appendices/June 2016 A19

Copyright of MIS Quarterly is the property of MIS Quarterly and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.