Source-linked AI summary
Toward a Perspectivist Turn in Ground Truthing for Predictive Computing
Valerio Basile, Federico Cabitza, Andrea Campagner, Michael Fell
TL;DR
The paper addresses the limitations of majority-aggregated gold standards in supervised ML, especially for complex or ambiguous phenomena. It proposes data perspectivism, which integrates or preserves human perspectives in ground truthing, and discusses its benefits, implementations, challenges, recommendations, and research agenda.
Problem
Majority-vote aggregation can be problematic when objects are complex, ambiguous, or classified differently by raters for reasons beyond error.
Method
The paper proposes weak and strong data perspectivist approaches that collect multiple perspectives, either summarizing them into single labels or retaining them through model training and benchmarking.
Results
The paper presents data perspectivism as an alternative ground-truthing paradigm and examines its potential advantages, implementations, disadvantages, recommendations, and research agenda.
Takeaways & Limitations
Data perspectivism aims to improve the representativeness and reliability of ML ground truth by addressing information richness, redundancy, uncertainty, inconsistency, and minority perspectives.
Takeaways & Limitations
Perspectivist ground truthing requires multiple raters, may be costly or time-consuming, can be incompatible with standard ML methods, and complicates validation without a unique ground truth.
Abstract
from arXiv · showhide
Most Artificial Intelligence applications are based on supervised machine learning (ML), which ultimately grounds on manually annotated data. The annotation process is often performed in terms of a majority vote and this has been proved to be often problematic, as highlighted by recent studies on the evaluation of ML models. In this article we describe and advocate for a different paradigm, which we call data perspectivism, which moves away from traditional gold standard datasets, towards the adoption of methods that integrate the opinions and perspectives of the human subjects involved in the knowledge representation step of ML processes. Drawing on previous works which inspired our proposal we describe the potential of our proposal for not only the more subjective tasks (e.g. those related to human language) but also to tasks commonly understood as objective (e.g. medical decision making), and present the main advantages of adopting a perspectivist stance in ML, as well as possible disadvantages, and various ways in which such a stance can be implemented in practice. Finally, we share a set of recommendations and outline a research agenda to advance the perspectivist stance in ML.
1 Motivations and Background
Supervised ML relies on manually annotated datasets whose labels are commonly aggregated by majority vote, but this can be problematic for complex, ambiguous, or expertise-dependent phenomena. The paper proposes integrating annotators’ perspectives more comprehensively instead of treating aggregation as only a data-quality issue.
- 1 Motivations and Background: Supervised ML datasets are manually annotated through classification schemas, data collection, annotation, and label aggregation.The standard pipeline associates each object with categorical labels and commonly concludes with aggregation.
- 1 Motivations and Background: Majority voting can fail when most raters misclassify complex objects or when the phenomenon is ambiguous, as often occurs in NLP.The paper links these problems to expert minorities and inherently ambiguous objects.
- 1 Motivations and Background: The proposed paradigm replaces monolithic, majority-aggregated gold standards with methods that more inclusively integrate human opinions and perspectives.The shift targets the knowledge-representation step of modeling processes.
2 Strong and weak data perspectivism
Data perspectivism distinguishes weak approaches, which consider multiple perspectives before producing a single label, from strong approaches, which retain multiple labels or data throughout training or benchmarking. Multiple ratings can represent population perceptions and disagreement, but they introduce practical and evaluative challenges.
- 2 Strong and weak data perspectivism: Weak perspectivism collects multiple raters and annotations, then may summarize them into single positions or labels.A discussion among experts or weighted majority voting can remain perspectivist if multiple opinions are considered.
- 2 Strong and weak data perspectivism: Perspectivist annotation can be motivated by distrust in raters or by recognizing that the phenomenon itself varies across observers.These motivations apply even when systems ultimately propose one best label.
- 2 Strong and weak data perspectivism: Multiple labels can capture disagreement arising from ambiguity, complexity, temporal instability, task difficulty, or legitimate differences between raters.The paper treats such variability as potentially inter-subjective rather than necessarily erroneous.
- 2 Strong and weak data perspectivism: Collecting multiple labels can produce a more representative sample of perceptions, opinions, and judgments, while a qualified majority may support higher accuracy.The text presents representativeness and qualified aggregation as distinct uses of multiple ratings.
- 2 Strong and weak data perspectivism: Strong perspectivism retains multiple labels or data for each object during subsequent model training or benchmarking.One backward-compatible implementation replicates each training object according to the number of times raters assigned a label.
3 Between objectivity and subjectivity
The paper replaces the value-laden opposition between objectivity and subjectivity with an intersubjectivity spectrum, where judgments vary in their shared interpretability. Locating a phenomenon on this spectrum can guide whether an ML pipeline should preserve multiple perspectives.
- 3 Between objectivity and subjectivity: Subjective phenomena are associated with high observer variability, commonly assessed using inter-rater agreement metrics such as kappa, alpha, or rho.The paper contrasts these phenomena with objective judgments, which most people would ideally classify similarly.
- 3 Between objectivity and subjectivity: Metrological accounts link dependable measurement to both relation to an object and interpersonal interpretability.This frames ground truthing as measurement while retaining attention to how different subjects interpret the result.
- 3 Between objectivity and subjectivity: Objectivity and subjectivity are treated as ideal extremes of a continuous intersubjectivity spectrum rather than as value-laden opposites.The spectrum describes how phenomena present themselves to interpretation and judgment within a human community.
- 3 Between objectivity and subjectivity: Neither total subjectivity nor full objectivity is attainable because judgments are socially situated and context-dependent.Shared cultural conventions prevent total subjectivity, while contingent conditions affect every judgment or measurement, including machine-based ones.
- 3 Between objectivity and subjectivity: Assessing intersubjectivity helps determine whether to adopt weak or strong perspectivism and whether retaining multiple labels preserves expression or undermines reliability.Highly intersubjective phenomena involve limited disagreement, whereas low-intersubjectivity cases involve complexity, ambiguity, or specialized competence.
4 Related work
Related ML communities have traditionally relied on single-rater or majority-aggregated gold standards, whose reliability and representativeness have been increasingly questioned because of observer variability. Recent work develops both weak and strong perspectivist methods that preserve or model disagreement during training and evaluation.
- 4 Related work: NLP, computer vision, and medical informatics have traditionally used gold-standard datasets built from one rater or majority aggregation of a few raters.The reliability and representativeness of these datasets were historically scarcely questioned.
- 4 Related work: Expanded raw-data availability and crowdsourcing have intensified attention to observer variability, including evidence that ImageNet majority labels can misrepresent its images.Observer variability was already recognized in computational linguistics and medical settings.
- 4 Related work: Many crowdsourced-learning studies adopt weak perspectivism by developing ML methods that account for variability among annotators.These approaches represent an established strand of perspectivist work.
- 4 Related work: Recent strong-perspectivist studies model individual raters alongside consensus and report improved classification accuracy in medical imaging.Other work reports performance increases from strong-perspectivist training and investigates disagreement in annotation.
- 4 Related work: Conceptual and empirical studies argue that annotation disagreement can reflect task subjectivity and should be systematically incorporated into model evaluation rather than discarded as noise.This work also examines strong perspectivism's effects on supervised-model development and evaluation.
5 Looking at the two sides of the same coin
Perspectivist ground truthing preserves classification multiplicity, offering benefits such as representing ambiguity, disagreement, minority views, and uncertainty. Its costs include additional annotation demands, incompatibility with standard ML methods, and more difficult validation without a unique ground truth.
- 5 Looking at the two sides of the same coin: Perspectivist ground truthing retains multiple classifications instead of collapsing them through majority voting or consensus surveys.The paper presents this preservation of multiplicity as having both advantages and shortcomings.
- 5 Looking at the two sides of the same coin: It provides a theoretical basis for treating categorical irreducibility as intrinsic to naturally ambiguous tasks, including NLP and inconsistent clinical manifestations.The approach recognizes that some phenomena cannot be reduced to a single categorical interpretation.
- 5 Looking at the two sides of the same coin: Disagreement can supply information for decision support in borderline and complex cases rather than being discarded as label noise.The approach also aims to represent minority views, human error patterns, label uncertainty, and fuzzy data.
- 5 Looking at the two sides of the same coin: Perspectivist methods can support three-way, fuzzy, and probabilistic outputs that represent subjectivity and uncertainty for decision makers.The paper associates these methods with trust, user experience, and possible mitigation of automation bias and deskilling.
- 5 Looking at the two sides of the same coin: Multiple-rater annotation can create substantial cost and time bottlenecks, especially in medicine or large datasets.This requirement may make perspectivist ground truthing impractical or expensive in some settings.
- 5 Looking at the two sides of the same coin: Standard ML systems generally cannot accommodate multiple perspectives directly, requiring ad-hoc methods whose usefulness in perspectivist ground truthing remains unclear.Some multi-label algorithms may handle multiple labels, but their applicability is not established.
- 5 Looking at the two sides of the same coin: Validation and evaluation become more complex because perspectivist settings lack a uniquely defined ground truth.Majority labels can serve as benchmarks in some cases but may be inappropriate for strongly subjective or ambiguous tasks.
6 Recommendations and a research agenda
The paper recommends preserving and evaluating multiple perspectives in ML ground truthing rather than reducing judgments to single labels, while identifying benchmarks and measures needed to advance this approach.
- Recommendations: Perspectivist ground truthing should preserve information richness, including redundancy, uncertainty, and inconsistency, instead of reducing multiple ratings to single labels.The paper frames this shift as an alternative to reductionist ground truthing and identifies recommendations and research directions for managing the resulting complexity.
- Recommendations: Annotation schemes can allow multiple labels, “none of these” responses, and judgments that the available label set is inadequate.These design choices directly acknowledge perspectives expressed by individual raters.
- Recommendations: Researchers should involve enough heterogeneous raters to support statistically significant or superhuman-accuracy majorities and to treat disagreement as potentially informative rather than merely noisy.The recommended heterogeneity spans origin, culture, expertise, and skills.
- Recommendations: ML models should be evaluated for robustness on external datasets and accompanied by detailed reporting of rater enrollment, expertise, incentives, instructions, and rating quality.Performance degradation outside the training setting may indicate overfitting to a non-representative user sample.
- Recommendations: Rater confidence and other judgment-level information can help identify difficult cases and enable reductions beyond simple majority voting.Suggested alternatives include weighted majority and strong versions of data perspectivism, especially for complex objects such as medical cases.
- A research agenda: The research agenda calls for benchmark datasets and better ways to represent the informational value of disagreement, which standard agreement or entropy measures do not fully capture.Benchmarks would support evaluation of perspectivist models across data types and classification tasks, while disagreement can be both rich and noisy.
- A research agenda: Data perspectivism targets the representativeness and reliability of ML ground truth by questioning single-truth assumptions and making room for minority opinions and ethical reflection.The paper also associates the approach with more accurate and explainable quantitative model evaluation and questions about whose opinions models rely on.