A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

• Prior research found workers often earned a fraction of the U.S. minimum wage and reported nonpayment for adequately completed tasks [35].Detailed worker-focused earnings research was limited by a lack of granular quantitative data; existing data-driven studies largely examined employers or market dynamics.Source: INTRODUCTION · p. 1
• Logs of 3.8 million submitted or returned HITs from 2,676 workers, collected between September 2014 and January 2017, enabled task-level wage estimates using task duration and reward.The records came from the Crowd Workers Chrome plugin.Source: INTRODUCTION · p. 1
• Workers’ mean and median hourly wages were $3.13/h and $1.77/h, and 111 workers (4%) earned more than $7.25/h.Source: INTRODUCTION · pp. 1–2
• Requesters paid $11.58/h on average, but unpaid work lowered worker wages: ignoring it raises estimated median and mean hourly wages to $3.18/h and $6.19/h, respectively.Unpaid work included searching for tasks, working on rejected tasks, and working on tasks that were not submitted.Source: INTRODUCTION · p. 2
• The study examines work and task-selection strategies that workers could use to raise hourly wages and discusses design opportunities to improve working environments on AMT and other crowdsourcing platforms.Source: INTRODUCTION · p. 2

BACKGROUND

• Crowdsourcing platforms may favor requesters through asymmetric ratings, limited worker recourse, pay-per-work, and contractor status that exempts requesters from minimum-wage obligations.These features contribute to earnings instability and stressful working conditions.Source: BACKGROUND · p. 2
• Workers typically earn a fraction of the U.S. minimum wage, and many report unpaid adequately completed tasks, despite income being their primary motivation [35].Low wages and ethical concerns also matter to requesters because poor working conditions are associated with lower quality and higher turnover.Source: BACKGROUND · p. 2
• Prior earnings research relied partly on Turker Nation conversations, which reveal workers’ beliefs but may not provide unbiased, representative earnings estimates.Detailed quantitative research had been limited by the absence of adequate data.Source: BACKGROUND · p. 2
• Existing quantitative research largely examined employers or market dynamics, leaving data-driven analysis of worker treatment missing; this study uses Crowd Workers Chrome plugin logs to evaluate hourly wages at scale.Prior work included pricing methods, task design [40], and market dynamics.Source: BACKGROUND · p. 2

TERMINOLOGY

• Launched in 2008, AMT is one of the largest micro-task sites; common tasks include transcription, data collection, image tagging, and classification, with audio transcription becoming more prevalent.Ipeirotis identified the common task types, and Difallah et al. reaffirmed them while noting the growing prevalence of audio transcription.Source: TERMINOLOGY · p. 2
• A standalone unit of work on AMT is a task or HIT, listed on platform webpages or, in some cases, external webpages.Source: TERMINOLOGY · p. 2
• Requesters issue HITs, may group concurrently completable tasks into a HIT group, and can require worker qualifications such as a >95% HIT approval rate.Source: TERMINOLOGY · p. 2
• Workers who meet qualifications can accept HITs; after completion, they submit work for requester approval or rejection, while an accepted but unfinished HIT is returned.Workers receive a financial reward for approved submissions.Source: TERMINOLOGY · p. 2

DATASET

• The opt-in Crowd Worker Chrome plugin tracked task acceptance, submission or return, timestamps, HIT metadata, and visits to predefined web domains; approval status was available for only 29.6% of HIT records.The plugin was designed to disclose effective hourly wages, following design suggestions in.Source: DATASET · pp. 2–3
• The logs omit fine-grained interactions and browsing outside predefined domains, and without HIT interface contents the researchers could not compute task accuracy.Fine-grained interactions such as keystrokes and mouse movements were not collected because they could contain personally identifiable information.Source: DATASET · p. 3
• 3,808,020 HIT records from Sept. 2014 to Jan. 2017 cover 2,676 workers, 104,939 HIT groups, and 20,286 requesters, making this the largest known AMT worker-log dataset for hourly wage analysis.Tracked HITs peaked at N=386,807 in June 2016; January 2017 data is partial, so analyses mainly reflect activity from late 2015 to late 2016.Source: DATASET · p. 3, Figure 1 · p. 3
• Workers averaged 1,302 HITs each (SD=4722.5; median=128.5) and 54.0 hours worked (SD=172.4; median=6.14h).Workers used the plugin for 69.6 days on average (SD=106.3; median=25 days); counts ranged from one HIT for 135 workers to 107,432 HITs for one worker.Source: DATASET · p. 3, Figure 2 · p. 3
• Filtering the top and bottom 5 percent of submitted HIT durations left N=3,471,580 HITs (91.2% of the original number).The filtered data represents 99,056 unique HIT groups, 2,666 workers, and 19,598 requesters.Source: DATASET · p. 3
• The dataset retains N=23,268 (0.7%) zero-reward HITs, typically qualification tasks, because completing them still counts as work.The authors state that this small portion does not significantly impact results.Source: DATASET · p. 3

Measuring the Hourly Wage

• Overlapping HIT intervals can lower interval-based wage estimates, while uncounted time between HITs can raise them; temporal clustering groups nearby HITs into work sessions.The method groups temporally close intervals using a threshold D, accounting for overlap and short gaps between tasks.Source: THE AMT WAGE DISTRIBUTION · p. 4

Hourly Wages per HIT/Cluster

• Task/cluster mean wage rates ranged from $4.80/h to $6.19/h; excluding $0 HITs, the interval-based median was $3.31/h and mean was $6.53/h.The distributions included 460,939 $0-paid HITs, which were qualification tasks and/or returned.Source: THE AMT WAGE DISTRIBUTION · p. 4, Figure 5 · p. 5
• At D=1min, per-cluster median and mean wages were $1.77/h and $4.80/h, decreases of $1.41 and $1.39 from D=0, respectively.At D=0, 61.1% of workers worked on two or more HITs in parallel, which raised estimated work-unit wages; the D=1 results show that unpaid intervals also materially affect estimates.Source: THE AMT WAGE DISTRIBUTION · p. 5, Figure 5 · p. 5

Hourly Wages per Worker

• Per-worker mean wages were $3.13–$3.48/h and medians $1.77–$2.11/h; only 4.2% of workers averaged above the federal minimum wage.Workers earned an average of $95.96 (SD=310.56; median=$11.90). The authors could not attribute higher hourly wages to platform experience alone and use the interval-based method in later analyses because it preserves task-level detail.Source: THE AMT WAGE DISTRIBUTION · p. 5, Table 2 · p. 5

Unpaid Work

• Returning HITs had the largest wage impact, time between HITs the second-largest, and rejection the smallest, though rejection risks future access or bans.Workers spent 45,778 hours on returned HITs and averaged $30.44 in lost time value; time between HITs totaled 4,602.7 hours, or 103.6 minutes per worker on average.Source: FACTORS AFFECTING THE HOURLY WAGE · pp. 5–6

HIT Reward

• HITs with rewards above $0.64 were projected to pay above the $7.25/h minimum wage, and higher-reward HITs generally yielded higher hourly wages.Some low-reward HITs also yielded high hourly wages, but workers may find it harder to identify them in advance.Source: FACTORS AFFECTING THE HOURLY WAGE · p. 7

Requesters

• Requesters paid an average of $11.58/h and a median of $4.57/h; higher-reward requesters were associated with higher hourly pay.The authors note that fair-paying requesters post significant numbers of HITs, so prioritizing their work could improve worker wages; tools could alert workers to their HITs.Source: FACTORS AFFECTING THE HOURLY WAGE · p. 7

Qualifications

• HITs requiring at least one qualification had median interval-based hourly wages of $2.63/h versus $2.45/h without requirements, while mean wages were $5.65/h (SD=19.1) versus $5.67/h (SD=28.2).Source: FACTORS AFFECTING THE HOURLY WAGE · p. 8

Types of Work

• Hourly wages varied by HIT topic: data/image transcription was low-paying, whereas “video evaluation” was high-paying, with a $10.30/h median wage among 49,720 HITs.Topics were identified through automated clustering of HIT documents and manual retrieval of recurring topical phrases.Source: FACTORS AFFECTING THE HOURLY WAGE · pp. 8, 10

DISCUSSION

• Unpaid task search, qualification tasks, and returned or rejected work depress hourly wages, and tools that help novices minimize this work could encode experienced workers’ learned strategies.Workers are not paid for these activities, and the discussion suggests that experienced workers learn to minimize such unpaid work over time.Source: DISCUSSION · p. 10
• 96% of workers earn below the U.S. federal minimum wage, while requesters pay $11.58/h on average; dominant requesters’ many low-wage HITs pull down the overall distribution.Because workers primarily seek income, low pay is especially concerning for people excluded from formal employment; image transcription, which may suit autistic workers, is among the lowest-paid work in the topical analysis.Source: DISCUSSION · p. 10
• Visualized wage feedback and real-time earnings strategies could improve task selection and work patterns; different wage-computation methods yielded estimates that were not largely different.Designers should consider privacy, and workers may be less likely to adopt the Crowd Worker plugin if its limited data collection is not visible to them.Source: DISCUSSION · p. 10
• Because most requesters pay below minimum wage, avoiding unfair requesters is insufficient; addressing the root problem may require better worker communication or higher minimum rewards.Fair requesters do not post HITs all the time, so avoiding them does not resolve the presence of unfair requesters.Source: DISCUSSION · p. 10
• Poor requester communication can leave broken HITs unrepaired and increase returned-task time, while better channels could also enable collective bargaining.The discussion notes that nudging workers to communicate or negotiate individually or collectively is difficult, but overcoming these barriers could benefit workers.Source: DISCUSSION · p. 10

• Experienced workers such as super turkers may have been less interested in the plugin, although the sample included 2,676 workers averaging 1.3k completed tasks each.Source: LIMITATIONS · p. 10
• Estimated work intervals may not reflect time spent actively working on HITs, a limitation shared by time-based payment schemes.Source: LIMITATIONS · p. 10
• The data did not capture scripts that may automate parts of workers’ tasks and increase their earnings, and the analyses do not establish causality.Source: LIMITATIONS · pp. 10–11
• The analysis lacked worker demographics; an online survey is planned, while investigation of ethical issues such as overtime remains future work.Source: LIMITATIONS · p. 11

CONCLUSION

• ~$2/h was the median hourly wage, and only 4% of workers earned more than $7.25/h.The median validates past self-report estimates, while the 4% figure underscores concerns about non-payment of the minimum wage.Source: CONCLUSION · p. 11
• The analysis used log data from 2,676 workers performing 3.8 million Amazon Mechanical Turk tasks.The study analyzed wages at the task level.Source: CONCLUSION · p. 11
• The study identified task search, task rejection, and task return as three sources of unpaid work that affect hourly wages.Source: CONCLUSION · p. 11
• The study explored task characteristics and working patterns associated with higher hourly wages.Source: CONCLUSION · p. 11