Source-linked AI summary
Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP
Yinuo Xu, David Jurgens
TL;DR
NLP annotation often collapses subjective or ambiguous judgments into majority-vote labels, potentially obscuring meaningful perspectives. This survey organizes disagreement sources and disagreement-aware models through prediction targets and pooling structures, finding a shift toward explicit and structured modeling while identifying limited distributional coverage and mostly descriptive fairness evaluation.
Problem
Majority-vote annotation can be problematic for subjective or ambiguous NLP tasks because it may obscure minority perspectives and produce representational harm.
Method
The survey synthesizes disagreement-aware NLP research using a taxonomy of data, task, and annotator factors and a framework based on prediction targets and pooling structures.
Results
The literature shifts from consensus prediction toward individual, group-aware, and population or subgroup distribution modeling, with increasing use of partial pooling.
Takeaways & Limitations
Disagreement is treated increasingly as meaningful structured variation, while future work should integrate multiple sources and develop interpretable, normatively grounded evaluation frameworks.
Takeaways & Limitations
The survey’s taxonomy is a simplification because data, task, and annotator factors cannot always be separated cleanly, and its review excludes some tangential approaches.
Abstract
from arXiv · showhide
Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.
1 Introduction
The survey challenges majority-vote ground truth for subjective or ambiguous NLP tasks and develops a unified, model-centric synthesis of disagreement-aware research.
- Motivation: Majority voting can obscure valid minority perspectives and contribute to biased models and representational harm in complex or subjective NLP tasks.The paper situates this concern in tasks such as toxic language detection and quality estimation.
- Contributions: The survey formalizes disagreement across data-, task-, and annotator-driven sources.
- Contributions: It synthesizes disagreement-aware methods by mapping them to disagreement sources, prediction targets, and pooling structures.
- Scope: The survey jointly examines NLP-specific subjectivity, modeling approaches, evaluation practices, and fairness considerations across more than 120 prior works.
2 Sources of Disagreement
The proposed taxonomy organizes disagreement into data, task, and annotator factors, while emphasizing that these sources can interact and produce systematic or inconsistent interpretations.
- Taxonomy: The taxonomy groups disagreement into data factors, task factors, and annotator factors.Examples include ambiguity and data quality; task formulation and interface design; and demographics, preferences, and errors.
- Interactions: Different sources interact, so linguistic ambiguity can yield interpretations that vary systematically with annotators’ individual and group identities.
- Data factors: Data quality issues, linguistic ambiguity, and epistemic uncertainty can make disagreement reflect irreducible multiplicity rather than annotation error.
- Annotator factors: Annotator identities, behavioral inconsistencies, and low-quality or strategic labeling introduce systematic or random variation into annotations.
- Task factors: Task formulation and interface design shape disagreement by permitting multiple interpretations or increasing inconsistency through unclear guidelines.
3 Learning from Disagreement
The survey traces a shift from latent-truth models that explain disagreement as annotator noise toward models that represent individual behavior and structured variation directly.
- Latent truth and annotator reliability: Latent-truth models infer one hidden label per item while attributing disagreement to annotator reliability, bias, task difficulty, or noise.
- Latent truth and annotator reliability: Partial pooling of annotator parameters can balance model expressivity and generalization in latent-truth approaches.
- Multi-annotator models: Multi-annotator models treat each annotator as a distinct task and learn predictions over annotator–item pairs instead of collapsing responses into one truth.
- Embedding-based models: Embedding-based models represent disagreement through interactions between item and annotator embeddings, supporting shared latent spaces and scalability to many annotators with sparse labels.
- Distributional models: Distributional approaches predict item-level disagreement distributions, combining alignment with individual annotations and divergence-based alignment to empirical distributions.
- Coverage and directions: Across the surveyed methods, coverage of data and task factors remains limited, motivating future work beyond annotator-centered modeling.
4 Mapping Models to Sources of Disagreement
The survey maps model families onto disagreement sources, showing that approaches differ in whether they encode item characteristics explicitly or implicitly while modeling annotator and task variation.
- Mapping model families: Latent-truth models span task factors, annotator factors, and approaches that condition annotator behavior on the input.
- Mapping model families: Within annotator-and-data modeling, implicit approaches use shared encoders or embeddings without explicitly representing item ambiguity as a disagreement source.
- Mapping dimensions: The mapping organizes methods by prediction targets and pooling assumptions, including consensus labels, annotator responses, group outputs, and full disagreement distributions.
5 Prediction Targets and Pooling Structures
The survey organizes disagreement-aware models by prediction target and pooling structure, revealing a shift from consensus outputs toward structured modeling of annotators and populations.
- Figure 3 maps methods by prediction target and pooling structure, covering consensus, annotator, group, and distribution predictions alongside group, unpooled, and partial pooling.Partial pooling represents structured relationships among annotators, populations, or subpopulations.
- Group pooling forces annotators to share one labeling function, whereas unpooled and partially pooled models represent individual behavior or structured relationships.
- The field has progressed from consensus prediction to annotator, group, and population or subgroup distribution modeling.
- Methods have also shifted from unpooled toward partial pooling, with hierarchical priors, inter-annotator kernels, demographic-informed mixtures, and subgroup-aware distributions highlighted as successful examples.
6 Evaluation Metrics
Evaluation of disagreement-aware models combines predictive metrics against annotations with measures of annotator behavior, but fairness assessment remains largely descriptive and some distributional metrics have problematic properties.
- Predictive evaluation ranges from hard metrics assuming one gold label to likelihood, ROC–AUC, density, entropy, per-annotator, and distributional measures.Per-annotator metrics include MAE, RMSE, and macro F1; distributional measures include KL and JSD.
- Cross Entropy and some divergence measures may violate symmetry or fair penalization when comparing models that reproduce disagreement distributions.
- Annotator-quality evaluation uses likelihood, annotator–model correlations, and agreement coefficients, although agreement metrics can conflate ambiguity with unreliability.
- Diagnostic frameworks report parity gaps without explicit normative grounding, motivating adaptation of statistical parity, equalized opportunity, and subgroup fairness.
7 Challenges and Future Work
The survey identifies unresolved challenges in simulating, explaining, and generalizing disagreement-aware models, alongside practical and normative tradeoffs in adopting perspectivist approaches.
- Persona- and demographic-conditioned LLM simulations are prompt-sensitive, largely unpooled, vulnerable to stereotypes, and limited in how much annotator variance they explain.Persona variables explain only a small fraction of variance, while demographic fine-tuning may rely on annotator-specific rather than generalizable group signals.
- Task-level disagreement from instructions, label schemas, presentation order, and interfaces remains underexplored, while richer disaggregated annotation data is needed.
- Interpretability methods rarely explain why annotators or groups disagree or validate explanations against real annotator behavior, especially in demographic-aware and mixture-of-experts models.
- Disagreement-aware models face dataset-specific generalization, annotation and computation costs, difficult filtering decisions, and normative questions about which perspectives to preserve.
8 Conclusion
The survey presents disagreement as meaningful variation rather than noise and unifies its sources, models, evaluation gaps, and future research directions.
- The survey introduces a taxonomy of data-, task-, and annotator-driven disagreement sources and traces modeling from aggregation to multi-annotator and persona-conditioned approaches.
- Its synthesis highlights limited distributional modeling and fairness evaluation that is largely descriptive rather than normatively grounded.
- Future directions include integrating multiple disagreement sources and developing interpretable, normatively grounded evaluation frameworks.
9 Limitations
The survey is not exhaustive and its taxonomy simplifies interactions among disagreement sources. Its synthesis also inherits a literature-wide limitation: observed disagreement may conflate meaningful perspectives with confusion or fatigue because annotation practice is rarely designed to elicit specific forms of disagreement.
- The survey reviews over 120 papers selected for methodological relevance, excluding some adjacent approaches such as LLM persona simulations.The authors note that a fully exhaustive survey is difficult and that selection criteria shaped coverage.
- The proposed taxonomy is a simplification because data, task, and annotator factors can interact and cannot always be separated cleanly.The authors identify extensions that integrate interactions among these factors as a future need.
- Observed disagreement may combine meaningful perspective variation with confusion or fatigue, limiting comparisons across papers.The literature often treats disagreement as an observed dataset property rather than an outcome of annotation-practice design.
A.1 Survey Scope and Selection
The survey focuses on NLP tasks where annotator disagreement is theoretically meaningful and reviews over 120 methodologically relevant works. It organizes this literature around disagreement sources, modeling families, prediction targets, pooling structures, and a unified technical and fairness-oriented synthesis.
- Survey Scope and Selection: The survey covers NLP settings such as toxicity, stance, and sentiment where disagreement reflects subjective or socially grounded judgments rather than primarily measurement error.Its scope is limited to research that studies, models, or evaluates annotator disagreement in text-based tasks.
- Survey Scope and Selection: The authors review over 120 works and include papers that propose learning algorithms, prediction targets, or evaluation frameworks accounting for multiple annotators or divergent interpretations.Papers may appear in multiple categories, and approximately 12 persona-prompting papers are discussed mainly under challenges and future work.
- Survey Scope and Selection: The survey positions itself as a technically grounded NLP synthesis combining modeling approaches, evaluation practices, and fairness implications.It complements conceptual and modality-general surveys by centering task design, interpretation practices, and annotator identities.
- Detailed Synthesis Table of Prediction Targets and Pooling Structures: The survey maps modeling families to prediction targets and pooling structures, including consensus, annotator, group, and distribution targets.Figure 4 organizes approaches by what they predict and how information is pooled across annotators.