Source-linked AI summary
CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation
Mark Diaz, Ian D. Kivlichan, Rachel Rosen, Dylan K. Baker, Razvan Amironesei, Vinodkumar Prabhakaran, Emily Denton
TL;DR
Ethical considerations in crowdsourced dataset annotation remain underexamined, despite the role of human-annotated data in ML. The paper synthesizes literature on annotator identities and platform relationships, then presents CrowdWorkSheets for documenting annotation decisions. It argues that annotators’ experiences can shape dataset labels and should be documented alongside dataset development.
Problem
Ethical considerations in the processes and decisions of crowdsourced dataset annotation have received insufficient attention, although human-annotated data is important to ML research and development.
Source: §1 · pp. 1–2Method
The paper synthesizes literature on annotators’ identities and platform relationships, then introduces CrowdWorkSheets to support reflection and transparent documentation of annotation decisions.
Source: §1 · pp. 1–2Results
The paper argues that annotators’ histories and experiences can become encoded in datasets, making annotators non-interchangeable in the annotation process.
Source: §6 · p. 9Takeaways & Limitations
Dataset developers should document annotator selection and work experiences alongside other aspects of dataset development.
Source: §6 · p. 9The case study’s disagreement analysis was preliminary and examined only a few annotator demographics, leaving other potential sources unanalyzed.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Human annotated data plays a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into dataset annotation have not received nearly enough attention. In this paper, we survey an array of literature that provides insights into ethical considerations around crowdsourced dataset annotation. We synthesize these insights, and lay out the challenges in this space along two layers: (1) who the annotator is, and how the annotators' lived experiences can impact their annotations, and (2) the relationship between the annotators and the crowdsourcing platforms, and what that relationship affords them. Finally, we introduce a novel framework, CrowdWorkSheets, for dataset developers to facilitate transparent documentation of key decisions points at various stages of the data annotation pipeline: task formulation, selection of annotators, platform and infrastructure choices, dataset analysis and evaluation, and dataset release and maintenance.
1 INTRODUCTION
Crowdsourced annotation supports modern ML, but the ethical choices shaping annotations and annotators’ experiences warrant attention. The paper surveys these concerns and introduces CrowdWorkSheets to document decisions throughout dataset annotation.
1 INTRODUCTION
- Crowdsourcing platforms efficiently distribute labeling microtasks, supporting the creation of training datasets and human-in-the-loop systems.Human computation uses human intelligence and cognition as elements in information-processing systems, often at large global scale.Source: §1 · p. 1
- The paper examines how annotators’ identities, social structures, and perspectives shape labels and the systems built from them.It challenges treating crowdworkers as interchangeable and notes that datasets continue to shape systems after workers finish their tasks.Source: §1 · pp. 1–2
- CrowdWorkSheets is a framework for critical reflection and transparent documentation of annotation decisions, processes, and outcomes, focused on worker subjectivity and experiences.It complements broader dataset documentation frameworks by addressing ethical problems specific to crowdsourced annotation.Source: §1 · p. 2
- The paper reviews literature on annotators’ social experiences and platform relationships, introduces documentation questions, and illustrates their use in a hypothetical case study.The case study follows how a dataset developer might document decisions.Source: §1 · p. 2
2 WHO IS ANNOTATING ML DATASETS AND WHY DOES IT MATTER?
Annotators’ social positions and lived experiences can affect how they interpret tasks and contribute to dataset labels. The section argues that these differences matter for representing communities and understanding what datasets capture.
2 WHO IS ANNOTATING ML DATASETS AND WHY DOES IT MATTER?
- Annotators are often treated as interchangeable, yet social position, identity, and experience shape their judgments and can go undocumented in datasets.For subjective tasks such as sentiment or hate-speech annotation, those judgments can shape the perspectives that models learn.Source: §2 · p. 2
2.1 Accounting for the socio-cultural backgrounds of annotators
- Different communities can interpret the same task differently, so disagreement may provide a valuable signal rather than merely noise.The authors stress that annotator identity and lived experience can affect responses; accounting for demographic skews also helps contextualize datasets and downstream use.Source: §2.1 · p. 2
2.2 Lived experiences of annotators as expertise
- Lived experience and situated expertise can inform annotations, while annotator backgrounds can also affect tasks that appear objective.Examples include gender-based harassment and local knowledge in annotations about gang-involved youth; medical experts may sometimes rely on knowledge beyond the text and identify relations erroneously.Source: §2.2 · p. 3
3 WORKER EXPERIENCES OF DATASET ANNOTATION
Workers’ pay, safety, and power relations are part of responsible dataset annotation, and can affect worker well-being and annotation conditions. The section describes constraints across compensation and working conditions, requester relationships, and collective action.
3 WORKER EXPERIENCES OF DATASET ANNOTATION
- Annotation work’s compensation, power imbalances, and structure can create barriers to worker well-being and the production of quality work.The authors frame these as considerations rooted in workers’ experiences of annotation itself.Source: §3 · p. 3
3.1 Compensation and working conditions
- Unpaid task-search labor and competition can reduce effective wages and contribute to long hours that conflict with crowdwork’s appeal of flexibility.Reported estimates include 18 minutes of unpaid work per paid hour and a median wage falling from $3.76 to $2.83 per hour when daily invisible labor is counted.Source: §3.1 · p. 3
- Content moderation can expose workers to disturbing material, while nondisclosure agreements and limited feedback channels can leave them with little recourse or support.The paper notes that the full long-term harm to workers’ well-being is difficult to assess.Source: §3.1 · p. 3
3.2 Power dynamics
- Distance and hierarchy can lead annotators to infer requesters’ expectations rather than resolve ambiguity, contributing to the portability trap when solutions move across social contexts.Workers also report rejected work and limited feedback; some platforms let requesters retain data even when work is rejected.Source: §3.2 · pp. 3–4
- Geographic blocking can let U.S. requesters determine which global perspectives are considered, while platform-moderated spaces have been ineffective at supporting worker organizing.Independent forums and tools such as Turkopticon let workers share requester information; Dynamo was designed to support collective action among AMT workers.Source: §3.2 · p. 4
4 CROWDWORKSHEETS: A DOCUMENTATION FRAMEWORK FOR CROWDSOURCED DATASET ANNOTATION
CrowdWorkSheets structures reflection and documentation across the crowdsourced annotation pipeline. Its questions address task design, annotator selection, platform conditions, disagreement analysis, and dataset maintenance.
4 CROWDWORKSHEETS: A DOCUMENTATION FRAMEWORK FOR CROWDSOURCED DATASET ANNOTATION
- CrowdWorkSheets organizes considerations and documentation questions across the dataset pipeline, from task formulation through dataset dissemination.The framework is intended to guide collection, use, and dissemination of crowdsourced annotations.Source: §4 · p. 4
4.1 Task formulation
- Task formulation should account for subjectivity, ambiguity, annotator assumptions and expertise, instruction clarity, and information disclosed before participation.The framework links task design to the intended use and downstream context, including whether lived experience should inform judgments.Source: §4.1 · pp. 4–5
4.2 Selecting annotators
- Annotators are not interchangeable: communities may label the same task differently, so pool choices should reflect intended use and consider inclusion risks and privacy.Sociodemographic data collection or publication can make annotators identifiable.Source: §4.2 · p. 5
4.3 Platform and infrastructure choices
- Platform choices shape pay, communication and feedback, worker support, and the flexibility to assemble an appropriate annotator pool.Requesters are also advised to consider living-wage standards and pay per hour when platforms set pay per item.Source: §4.3 · p. 6
4.4 Dataset analysis and evaluation
- Majority aggregation can obscure valuable disagreement and minority perspectives, so developers should consider preserving uncertainty and analyzing group differences.Disagreement may help dataset users understand how diverse perspectives are represented.Source: §4.4 · p. 6
4.5 Dataset release and maintenance
- Dataset usefulness can change as language, social conditions, legal definitions, or medical standards change, making temporal context and maintenance plans important.The framework prompts developers to consider conditions that may make annotations outdated or less useful.Source: §4.5 · p. 7
5 CASE STUDY
The hypothetical case study applies CrowdWorkSheets to an English-language toxicity dataset, documenting annotator selection and collection choices. Preliminary analysis found low overall agreement alongside demographic differences, and the authors note that social views may shift over time.
Case study design
- The hypothetical dataset comprises 20,000 English-language tweets, independently labeled on a four-point toxicity scale, with annotators selected partly for harassment experience and demographic representation.The pool aimed for gender balance and at least 10% annotators older than 65; it was disproportionately composed of women and people of color compared with platform demographics.Source: §5 · pp. 7–8
Platform and infrastructure choices
- The chosen platform was described as supporting living-wage pay, worker feedback, and screening, but meeting pool requirements came at the cost of limiting the number of tweets labeled.The study aimed for at least $25 per hour and paid $6.25 per 40-tweet batch designed to take no more than 15 minutes.Source: §5 · p. 8
Results
- Batches with at least two incorrect answers on five data-quality questions were removed, accounting for 12% of the data.The checks covered attention, self-consistency, and alignment with predefined ratings.Source: §5 · pp. 8–9
- Fleiss’ κ= 0.25 across raters, yet some demographic groups showed high agreement while differing significantly from the majority opinion.The authors interpret this as not necessarily indicating poor data quality.Source: §5 · p. 9
Dataset analysis and evaluation
- The authors’ preliminary disagreement analysis covered only a few annotator demographics, leaving other possible sources such as tweet topic and response speed unexamined.They frame the work primarily as data collection rather than analysis.Source: §5 · p. 9
Dataset release and maintenance
- The authors expect perceptions of tweets to change over time, so the dataset may not capture future shifts in views about language targeting groups considered marginalized later.They include tweet writing and annotation month and year as metadata and describe a possible process for appending future annotations.Source: §5 · p. 9
6 CONCLUSION
CrowdWorkSheets challenges treating annotators as interchangeable, arguing their histories and experiences can shape datasets. It offers dataset developers a framework and recommendations for documenting key decisions, while recognizing institutions’ role in shifting incentives toward transparent documentation.
6 CONCLUSION
- Annotators’ histories and experiences bring unique perspectives that can become encoded in datasets, so their selection and work experience should be documented.Source: §6 · p. 9
- CrowdWorkSheets provides dataset developers a framework for reflecting on and documenting key decisions in crowdsourced dataset development, alongside recommendations.Source: §6 · p. 9
- Large institutions can help shift incentives toward transparent dataset documentation, including through conference submission and reviewer guidelines.Source: §6 · p. 9