Quantifying the Invisible Labor in Crowd Work

1 INTRODUCTION

• Invisible labor includes unpaid work to find tasks, figure out how to complete them, or manage payments, activities workers undertake to complete paid work.The authors frame fair compensation as covering both paid tasks and this additional labor.Source: §1 · p. 1
• The study asks how much time workers spend on invisible labor and how that time affects their overall hourly wages.Source: §1 · p. 1
• 33% of workers’ daily MTurk time went to invisible labor, lowering median hourly wage from $3.76 to $2.83.The study followed 100 workers completing 40,903 tasks.Source: §1 · p. 2
• Payment management was the most time-consuming and common invisible-labor category; hyper-vigilant searching and monitoring requesters was second most time-consuming.Workers spent a median 4.5 minutes daily on payment-related timeouts, and 97% visited the earnings section.Source: §1 · p. 2

2 RELATED WORK

• Prior work on invisible labor and methods for measuring MTurk work time form the paper’s two main research foundations.Source: §2 · p. 2

2.1 Invisible Work

• Invisible labor has been described as paid-employment activities that workers perform in response to employer requirements but that are often overlooked or devalued.Earlier accounts also discuss how technology companies shift digitization tasks to workers and consumers.Source: §2.1 · pp. 2–3

2.2 Invisible Labor in Crowd Work

• Research on crowd work describes invisible labor as work shifted onto workers and notes that platforms could potentially reduce search time or pay for search and message-reading.The paper focuses on quantifying categories identified in prior crowd-worker research.Source: §2.2 · p. 3

2.3 Quantifying Working Time

• The study builds on plugin-based MTurk time measurement and wage studies, extending prior methods to classify and measure unpaid work.The authors report that their $3.76 average hourly estimate when excluding unpaid work roughly coincides with an earlier $3.18 estimate.Source: §2.3 · pp. 3–4

3.1.1 Methods for Quantifying Paid Labor.

• The plugin records HIT acceptance and completion, active time on HIT tabs, and daily earnings; the background process also accounts for multitasking and excludes multi-day batches.Multi-day batches and HITs comprised 0.6% of the sample and were filtered out.Source: §3.1.1 · p. 4, §3.1.4 · p. 7

3 METHODS

• The measurement methods deliberately err toward undercounting invisible labor, including excluding disputed activities such as reading instructions and breaks.The plugin detects these activities but counts them as neither paid nor invisible labor.Source: §3 · p. 4, §3.1.2 · p. 5, §3.1.3 · p. 7

3.1 Computational Mechanisms to Measure Invisible & Paid Labor

• A Chrome Extension combines a page crawler and background process to detect MTurk activity and measure time spent on paid and unpaid work.Source: §3.1 · p. 4, §3.1.2 · p. 5

3.1.2 Methods for Quantifying Invisible Labor.

• For invisible labor, the plugin tracks active time on other MTurk pages without recording what workers type, and uses conservative classifications.It detects page activity such as searching or messaging through browser events and page interaction.Source: §3.1.2 · p. 5

3.1.3 Quantifying Types of Invisible Labor.

• The study groups invisible labor into hypervigilance, lack of guidance, payments, and general logistics, using prior interview research to guide its categories.Examples include watching requester profiles, seeking guidance, checking payments, and logging into MTurk.Source: §3.1.3 · pp. 6–7
• The plugin detects category-specific activity through MTurk pages, including requester profiles, task searches, queues, messaging, HIT previews, and earnings pages.Source: §3.1.3 · pp. 6–7

3.2 Field Study

• The field study measured daily invisible and paid work during one week, including weekends, after recruiting workers through MTurk and Turker mailing lists.The initial survey asked about workers’ estimated invisible-labor time and demographics.Source: §3.2 · p. 8, §3.2.1 · p. 8

3.2.1 Field Study Logistics.

• Participants used the plugin while working normally, reviewed daily activity graphs, and later rated its tracking accuracy at a median of 4 out of 5.Each participant received $10 for taking part.Source: §3.2.1 · p. 8

4 RESULTS

• Workers spent a median of 33% of their daily MTurk time on invisible labor: 33 minutes, in addition to 1 hour 7 minutes completing HITs.Some workers spent as much as 5 hours and 31 minutes daily on invisible labor.Source: §4 · pp. 9–10
• HIT completion times had a long tail weighted toward tasks under one minute; time spent working and invisible labor were positively correlated (Spearman’s ρ = 0.283, p = 0.004).The correlation between percentage of time working and time in invisible labor was 0.517 (p = 0.000).Source: §4 · p. 10

4.0.1 Quantifying Invisible Labor and its Economic Costs.

• Including invisible labor, workers’ median hourly wage fell from $3.76 to $2.83.The $3.76 figure excludes invisible labor; the $2.83 figure includes it.Source: §4.0.1 · p. 11

4.0.2 Invisible Work for Different Segments of Workers.

• Masters spent 20% of work time unpaid versus 43% for non-Masters, a suggestive difference given only 21 Masters; invisible-labor time also differed by gender, tool use, and English-speaking status.Differences were not significant for disability, multitasking, or U.S.-versus-India groups.Source: §4.0.2 · p. 13
• Tool users spent 30 minutes more on paid work and earned $1.3 more daily; workers completing HITs in batches did 9.6% less invisible work.The authors suggest batches may reduce task-search and context-switching time.Source: §4.0.2 · pp. 13–14

4.0.3 Quantifying Categories of Invisible Labor.

• Payments led category time and were common: 97% checked earnings, while timeouts took a median 4.5 minutes; Hypervigilance followed at 11 minutes and Lack of Guidance at six.Timeouts affected 37% of workers; 92% started HITs they later returned.Source: §4.0.3 · pp. 14–15

4.1 Perceptions of Invisible Labor

• Workers estimated a median two hours daily on invisible labor and generally overestimated their time; 63% felt they spent too much or far too much time on it.The median relative error was −0.14; 27% felt they spent the right amount of time.Source: §4.1 · pp. 16–17

5 DISCUSSION

• 33 minutes of daily MTurk time spent on invisible labor corresponded to a drop in median hourly wage from $3.76 to $2.83.The authors describe these estimates as a lower bound because they used conservative measurement methods.Source: §5 · p. 17

5.0.1 Most Common and Most Time-Consuming Invisible Labor.

• 97% of workers checked their earnings dashboard at least daily, and payment-related activity was both the most common and most time-consuming invisible labor category.The discussion links these checks to verifying payment and monitoring daily income; interfaces showing current earnings could help, but might also create stress.Source: §5.0.1 · pp. 17–18

5.0.2 Second Most Time-Consuming Category of Invisible Labor.

• Hypervigilance took a median of 11 minutes daily; workers searched for tasks and stayed on-call, while platforms may use high-return cues to encourage vigilance.The authors suggest computational task assignment or recommendations could reduce search time, and tools could help workers assess likely wages in particular time windows.Source: §5.0.2 · p. 18

5.0.3 Invisible Labor in Other Workplaces and Policy.

• The discussion connects making unpaid labor visible to policy change, noting that including unpaid work in official statistics has helped policymakers attend to it and design responses.The authors propose tools to help citizens time data releases, identify receptive policymakers, and build public support.Source: §5.0.3 · p. 19

5.0.4 Design Implications & Future Work.

• The plugin could be adapted to other digital labor platforms by remapping which website activity counts as paid or unpaid, supporting cross-platform study and auditing.The authors also suggest visualizations for requesters, while cautioning that these could encourage unhealthy competition or enable surveillance.Source: §5.0.4 · p. 20

5.0.5 Critical Theory and Design to Address Invisible Labor in Different Digital Workplaces.

• The authors caution that interfaces may make invisible labor more bearable without addressing structural problems, and propose worker co-design and fictional narratives to imagine alternatives.They argue critical analysis is needed when designing interfaces intended to address invisible labor.Source: §5.0.5 · pp. 20–21
• The study’s insights are limited by its methodology and population, and it prioritizes breadth over depth; questions about experience, wages, multitasking, and task switching remain open.The authors identify longitudinal study as a future direction and say further analysis of worker experience is needed.Source: §5.0.4 · p. 19, §5.0.6 · p. 21

• $2.83 versus $3.76: accounting for invisible-labor time lowers workers’ hourly wages from $3.76 to $2.83.Source: §6 · p. 21
• Payments and hyper-vigilance were the two most time-consuming invisible-labor categories.Source: §6 · p. 21
• Workers tended to overestimate how much invisible labor they believed they performed.Source: §6 · p. 21
• The amount of invisible labor varied with a wide range of worker dynamics that the authors say deserve more investigation.Source: §6 · p. 21
• The authors hope their plugin will inspire audits of digital labor platforms and potentially generate positive policy innovations in digital work.Source: §6 · p. 21