The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

1 Introduction

• Annotator disagreement can improve model performance and calibration, surface minority voices, and uncover task ambiguities (Fornaciari et al., 2021; Baan et al., 2022; Prabhakaran et al., 2021; Balagopalan et al., 2023; Parrish et al., 2023).Source: §1 · p. 1
• The paper calls the emerging research treating disagreement as a paradigm shift the perspectivist turn.Source: §1 · p. 1
• It examines assumptions and practical challenges across paradigms, identifies unresolved normative questions, and offers recommendations and future research directions.Source: §1 · p. 1

2 The Longstanding Paradigm

• The longstanding paradigm aggregates multiple labels to approximate underlying ground truth, whereas perspectivist work treats variation as meaningful information.Source: §2 · p. 1

2.1 What Causes Disagreement?

• Disagreement has often been attributed to ambiguous tasks or low-quality annotators, while multiple labels and agreement measures serve as quality checks.Source: §2.1 · p. 2
• Equating statistical deviation from the mean with societal bias can dismiss minority views, although an affected annotator may have more relevant context than the majority.Source: §2.1 · p. 2
• Lived experience and community membership shape judgments, so annotators with different backgrounds can disagree without being low quality.Source: §2.1 · p. 2
• Task context and framing affect judgments, and over half of surveyed annotators wanted more context while over two-thirds thought knowing the task purpose would help.Source: §2.1 · pp. 2–3
• Disagreement also arises in seemingly clear-cut tasks, so no clear set of tasks can be assumed free of subjectivity or disagreement.Source: §2.1 · p. 3

2.2 Practical Challenges under the Longstanding Paradigm

• Diverse recruitment alone does not make labels representative: crowdworker pools can differ from populations of interest, and sparse assignments or prolific annotators skew whose views enter the data.Source: §2.2 · p. 3
• Majority aggregation discards minority opinions and can shift estimates from the population mean, while averaged labels may underrepresent minority views and leave models miscalibrated.Source: §2.2 · p. 3

3 The Perspectivist Turn

• Perspectivist approaches model individual labels, label distributions, or annotator behavior, collect more annotator input, and investigate disagreement rather than treating it only as noise.The paper examines how these approaches conceptualize disagreement and the practical and normative challenges that arise.Source: §3 · p. 4

3.1 Rethinking Causes of Disagreement

• Demographic differences matter, but they explain only part of disagreement; task-specific opinions and other experiences can also shape labels.The paper notes that several demographic attributes do not consistently predict disagreement across tasks, while opinions about online toxic content predict hate-speech labels and language differences can affect image labeling.Source: §3.1 · pp. 4–5

3.2 Emerging Practical Challenges

• Capturing disagreement while maintaining data quality remains difficult because removing noise can also discard meaningful opinions.The paper identifies clear-cut control samples and other quality checks as promising ways to distinguish inattentive labeling from disagreement worth preserving.Source: §3.2 · p. 5
• Evaluation remains a bottleneck: perspectivist papers often compare models with majority-vote labels, and the field lacks methods to evaluate single outputs against richer annotator data.The paper says this continued reliance on averaged gold labels undercuts the potential utility of perspectivist methods.Source: §3.2 · p. 5
• Collecting detailed opinions can burden minoritized groups and create a tradeoff between richer data and participants’ privacy.Potential responses include learning from less data, privacy-preserving methods, and community-led data ownership.Source: §3.2 · p. 5
• Institutional pressure for fast data collection can conflict with participatory work, which seeks reciprocal relationships and engagement with context and power dynamics.The paper calls for lowering participation barriers and incentivizing slower, more context-specific collection.Source: §3.2 · pp. 5–6

3.3 Emerging Normative Challenges

• Researchers should make explicit whose perspectives define a correct answer and how those perspectives are weighted, since unstated choices can cause harm or neglect minoritized views.The paper argues that deciding what machinery should model and evaluate requires first addressing these normative questions.Source: §3.3 · p. 6
• Tasks need explicit, context-specific bounds for acceptable variation, and personalization changes rather than eliminates normative decisions about appropriate use.Unstated boundaries for concepts such as toxicity can be difficult to contest; personalization may also support misinformation a user believes.Source: §3.3 · pp. 6–7

Before data labeling begins

• Before labeling, set acceptable bounds of disagreement explicitly and consider task-specific sources such as sociocultural differences, ambiguous labels, and differing opinions.The paper says these choices can help capture important differences while minimizing confusion.Source: §4 · p. 7

Recruitment

• Recruit annotators to match the intended population or expertise, and avoid filtering people merely because they disagree with others.Suggested alternatives include representative recruitment, stratifying underrepresented groups, checking intra-annotator agreement, and qualification tasks.Source: §4 · p. 7

Data labeling design

• Give annotators context about the data’s use and system effects, and use disagreement to revise instructions, label options, and feedback channels.The paper recommends ways to express uncertainty when labels are ambiguous or annotators offer unforeseen responses.Source: §4 · p. 7

Dataset documentation

• Document annotator selection, workload, filtering, normative bounds, and reasons for discarding data; retain individual labels when possible.Providing non-aggregated labels can avoid information loss from aggregation.Source: §4 · p. 7

Model design and evaluation

• Evaluate beyond aggregate-label accuracy by comparing label distributions, modeling individual annotators, and measuring calibration to population uncertainty.The paper gives KL divergence, cosine similarity, and correlation as possible distributional measures; evaluator disagreement can also help identify model weaknesses or disparate service quality.Source: §4 · pp. 7–8

Replace implicit normative decisions with explicit ones

• Choose explicitly where a task lies between equal-weight majority voting and expert-driven labeling, including whether lived experience counts as expertise.The appropriate point on that spectrum depends on the task; explicit choices can make decision rules easier to define and contest.Source: §5 · p. 8

Science and technology studies

• Research on classification and technology’s social effects can help trace how individual voices are retained, merged, or lost in systems.The paper also points to fairness and ethics critiques that question whether representative datasets alone address more fundamental problems such as surveillance.Source: §5 · p. 8

Voting and social choice

• Social choice research may inform preference aggregation, though annotation often cannot solicit the full population and can decide output qualities independently.The paper contrasts annotation with electoral settings, where the full population may be consulted and single decisions have widespread effects.Source: §5 · pp. 8–9

Pragmatics

• Pragmatics offers a way to examine how annotators’ assumptions about speakers, purpose, and audience shape interpretation.The paper connects these assumptions to demographic attributes, professions, speech situations, languages, and imagined audiences.Source: §5 · p. 9

Participatory design

• Participatory design can help practitioners examine power dynamics and assumptions embedded in their disagreement practices.The paper highlights traditions that give non-expert stakeholders direct input into technology design.Source: §5 · p. 9

Take advantage of nuanced output spaces to meet diverse stakeholder needs

• Treat preferences as potentially multidimensional: a single output may satisfy people who favor different, compatible qualities.The paper’s example is an output that is both non-discriminatory and concise, which could meet distinct preferences.Source: §5 · p. 9

• Longstanding data labeling commonly assumes tasks have a ground truth, uses majority-vote aggregation, and avoids a normative stance.Source: §6 · p. 9
• Perspectivist literature recognizes that datasets and models must account for the full spectrum of human perspectives.Source: §6 · p. 9
• The paper recommends considering causes of disagreement beyond demographics, tensions with data quality and research pressures, and normative considerations.Source: §6 · p. 9

• The paper is an analysis of key questions about data-collection paradigms, not a comprehensive meta-analysis or literature review.Source: Limitations and Ethical Considerations · p. 9
• Because the authors did not comprehensively search all papers on these issues, some relevant work may have been overlooked.Source: Limitations and Ethical Considerations · p. 9
• Overlooked work could narrow the perspectives considered in future research based on the paper’s proposed avenues.Source: Limitations and Ethical Considerations · p. 9