Source-linked AI summary
The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation
Barbara Plank
TL;DR
The paper addresses the gap created by treating human label variation as noise and assuming a single ground truth, despite variation across annotation tasks. It synthesizes definitions and research across data, modeling, and evaluation, provides a repository of un-aggregated-label datasets, and argues for a comprehensive interdisciplinary treatment. Its supported conclusion is that human label variation should be treated as an opportunity for progress, while the paper and repository remain necessarily incomplete.
Problem
Research commonly assumes a single ground truth, while human label variation affects data, modeling, and evaluation and existing work is fragmented across disciplines.
Method
The paper reconciles notions of human label variation, synthesizes approaches across the pipeline, identifies open issues, and provides a repository of datasets with un-aggregated labels.
Results
The paper concludes that human label variation is an opportunity rather than a problem and calls for comprehensive treatment across data, modeling, and evaluation.
Takeaways & Limitations
Future research should consider human label variation across the full pipeline and support open, interdisciplinary discussion using richer datasets and evaluation beyond hard labels.
Takeaways & Limitations
The position paper is necessarily incomplete because it synthesizes a broad notion and its dataset repository is also incomplete.
Abstract
from arXiv · showhide
Human variation in labeling is often considered noise. Annotation projects for machine learning (ML) aim at minimizing human label variation, with the assumption to maximize data quality and in turn optimize and maximize machine learning metrics. However, this conventional practice assumes that there exists a ground truth, and neglects that there exists genuine human variation in labeling due to disagreement, subjectivity in annotation or multiple plausible answers. In this position paper, we argue that this big open problem of human label variation persists and critically needs more attention to move our field forward. This is because human label variation impacts all stages of the ML pipeline: data, modeling and evaluation. However, few works consider all of these dimensions jointly; and existing research is fragmented. We reconcile different previously proposed notions of human label variation, provide a repository of publicly-available datasets with un-aggregated labels, depict approaches proposed so far, identify gaps and suggest ways forward. As datasets are becoming increasingly available, we hope that this synthesized view on the 'problem' will lead to an open discussion on possible strategies to devise fundamentally new directions.
1 Introduction
The paper challenges the conventional reliance on a single aggregated gold label, arguing that plausible human label variation is common and affects the data, modeling, and evaluation pipeline.
- Annotated datasets typically aggregate human labels, often by majority vote, into a single gold label for learning systems.
- Ground truth assumptions fit high-agreement questions but often fail for subjective, ambiguous, difficult, or multiply plausible language tasks.
- Human label variation can reflect genuine disagreement, subjectivity, difficult cases, or multiple plausible answers rather than annotation error alone.
- Treating variation as a single ground truth shapes data construction, model outputs, and evaluation around one preferred answer.
- The paper argues that neglecting human label variation is problematic because it impacts every stage of the machine-learning pipeline.
2 Data and Human Label Variation
The paper defines human label variation as plausible annotation variation arising from ambiguity, uncertainty, disagreement, or multiple correct options. It advocates preserving annotator-level labels and metadata to study richer data quality and modeling possibilities.
- Why variation matters: Irreconcilable annotator disagreement is observed across almost all NLP and computer-vision tasks, challenging reliability definitions that assume a fixed labeling standard.
- Defining human label variation: Human label variation denotes plausible annotation differences, distinguishing them from errors caused by attention slips or other annotation mistakes.
- Defining human label variation: HLV may arise from instance ambiguity, annotator uncertainty, genuine disagreement, or multiple correct options, while aggregation obscures this complexity.
- Related perspectives: The literature treats related phenomena as human uncertainty, contested labels, hard cases, difficult linguistic cases, or acceptable annotation ranges.
- Open issues and suggestions: Progress requires collecting and releasing annotator-level un-aggregated labels, documenting dataset creation, and including extensive metadata.
- Open issues and suggestions: The paper provides a repository overview of datasets with multiple annotations and encourages responsible release of annotator backgrounds, annotation processes, and instance-level metadata.
- Open issues and suggestions: Richer datasets may reveal model capabilities and limitations, support learning from fewer but more informative sources, and motivate revised agreement measures.
3 Modeling and Human Label Variation
Research on learning with human label variation is growing but fragmented across disciplines and methods. Proposed approaches either resolve variation into a single label or embrace richer annotation information during learning.
- Methods for handling human label variation fall into two camps: resolving variation and embracing it.The paper connects these approaches to emerging literature and identifies adoption gaps.
- Aggregation and filtering treat human label variation as noise and produce a presumed ground-truth label, with aggregation remaining the most widely adopted solution.
- Un-aggregated labels support methods including cost-sensitive weighting, multi-task learning, sequential fine-tuning, confusion matrices, annotator-level tasks, and soft-label tasks.
- Existing HLV research is fragmented across subjectivity, natural language inference, and joint NLP-CV work, leaving comprehensive evaluation and method transferability open.
- Learning from HLV depends on multiple-annotator data, whose availability may be limited; the balance between dataset size and annotator count remains unresolved.
- Separating annotation errors from meaningful signal is largely overlooked and requires theoretical work distinguishing errors from hard cases.
- Calibration to human majority labels can be theoretically and empirically problematic, motivating instance-level measures that better capture human label distributions.
- Data maps show that training on ambiguous instances can improve out-of-distribution generalization, while boundary cases can aid active learning.
4 Evaluation and Human Label Variation
Evaluation commonly relies on accuracy against a single gold label, which can obscure model behavior when human judgments vary. The paper surveys evaluation approaches that compare models with richer human-label information.
- Current evaluation typically uses accuracy against a gold standard, a practice the paper argues is severely flawed for many tasks.
- Many methods designed for human label variation are nevertheless evaluated against hard labels, so evaluation must move beyond accuracy.
- Soft-label evaluation compares the human label distribution with model outputs rather than considering only the top label.
- Proposed soft metrics include cross entropy, entropy correlation, and Kullback-Leibler or Jensen-Shannon divergence.
- Other evaluations compare models with individual annotators or use splits based on agreement, annotator clusters, item difficulty, or uncertainty flags.
5 Conclusions
The paper argues that human label variation affects data, modeling, and evaluation and should be treated as an opportunity rather than only a problem. It calls for comprehensive, interdisciplinary work and supports this agenda with a released dataset repository.
- Human label variation impacts every stage of the traditional machine-learning pipeline and is presented as an opportunity for innovation.
- The paper calls for a comprehensive treatment of HLV across data, modeling, and evaluation, supported by open interdisciplinary discussion.
- The released repository contains datasets with un-aggregated labels and is intended as a starting point for further research and contributions.
Limitations
The position paper synthesizes the broad notion of human label variation across learning from annotated data, but acknowledges that both the paper and its dataset repository are necessarily incomplete.
- The paper is necessarily incomplete because it aims to synthesize a very broad notion affecting all steps of learning from annotated data.
- The authors hope the paper and repository will stimulate open discussion and community uptake around human label variation.
Ethics Statement
Human label variation is linked to social bias because annotator backgrounds influence machine learning and evaluation, creating both risks and opportunities for representation.
- Annotator backgrounds influence annotations and consequently affect both machine learning and evaluation through social bias.
- Amplifying some groups’ voices can harm others because technologies involving human label variation have potential for dual use.
- Modeling human label variation can include underrepresented groups whose perspectives would otherwise be marginalized by majority views.
A Datasets with Multiple Annotations
The paper provides an overview of publicly available datasets containing human label variation data and records their use in broader empirical evaluations and shared tasks.
- The overview catalogs publicly available datasets containing human label variation data.
- Checkmarks indicate whether each source was used in surveys, listed in repositories, or included in shared tasks and empirical studies.