PaperlayerView exact revision on arXiv

Source-linked AI summary

LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task

Elisa Leonardelli, Silvia Casola, Siyao Peng, Giulia Rizzi, Valerio Basile, Elisabetta Fersini, Diego Frassinelli, Hyewon Jang, Maja Pavlovic, Barbara Plank, Massimo Poesio

arXiv:2510.08460v2cs.CL

TL;DR

Disagreement in human judgments is difficult to capture when models assume a single interpretation. LeWiDi-2025 expands the benchmark and evaluates both population-level distributions and individual annotator perspectives; LLM-based in-context-learning systems led the overall rankings, while fine-tuned transformers remained competitive.

  • Problem

    Many NLP tasks depend on subjective judgments, making the traditional practice of reconciling disagreements unsuitable for many such tasks and motivating ways to preserve and model variation.

    Source: §1 · p. 1

  • Method

    LeWiDi-2025 extends the benchmark across four datasets and evaluates population-level soft labels and individual annotator predictions using complementary metrics.

    Source: §1 · p. 2

  • Results

    LLM-based in-context-learning systems took the top overall leaderboard positions, with OCP-ICL and DeMeVa first and second; fine-tuned transformers remained competitive, and twinhter outperformed LLMs on smaller Par and VEN datasets.

    Source: §6.2 · p. 7

  • Takeaways & Limitations

    The task and released datasets are intended to stimulate further research, while the appropriate evaluation of ordinal human judgments remains an open issue.

    Source: §7 · p. 9

    The task covered text only and did not directly test generalization to unseen annotators, since test-set annotators were also present during training and development.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Many researchers have reached the conclusion that AI models should be trained to be aware of the possibility of variation and disagreement in human judgments, and evaluated as per their ability to recognize such variation. The LEWIDI series of shared tasks on Learning With Disagreements was established to promote this approach to training and evaluating AI models, by making suitable datasets more accessible and by developing evaluation methods. The third edition of the task builds on this goal by extending the LEWIDI benchmark to four datasets spanning paraphrase identification, irony detection, sarcasm detection, and natural language inference, with labeling schemes that include not only categorical judgments as in previous editions, but ordinal judgments as well. Another novelty is that we adopt two complementary paradigms to evaluate disagreement-aware systems: the soft-label approach, in which models predict population-level distributions of judgments, and the perspectivist approach, in which models predict the interpretations of individual annotators. Crucially, we moved beyond standard metrics such as cross-entropy, and tested new evaluation metrics for the two paradigms. The task attracted diverse participation, and the results provide insights into the strengths and limitations of methods to modeling variation. Together, these contributions strengthen LEWIDI as a framework and provide new resources, benchmarks, and findings to support the development of disagreement-aware technologies.

1 Introduction

LeWiDi-2025 extends the shared task’s effort to preserve and evaluate disagreement in human judgments. It broadens the benchmark and introduces complementary evaluation paradigms for population-level and individual-annotator predictions.

1 Introduction

  • The task responds to the view that reconciling differing subjective interpretations is not tenable for many NLP tasks, motivating research on preserving disagreement and modeling it.Researchers have also investigated how disagreement-rich corpora can train models and how such models should be evaluated.Source: §1 · p. 1
  • The third edition builds on earlier shared tasks, whose scope expanded from a benchmark including image classification to textual subjective NLP tasks and a second language.The first edition drew more than 100 benchmark downloads but had limited evaluation participation; the second attracted over 130 registrations, 30 prediction submissions, and 13 system papers.Source: §1 · p. 2
  • LeWiDi-2025 adds NLI, irony, conversational sarcasm, and paraphrase detection, with soft-label and perspectivist evaluation and new metrics for both paradigms.Two datasets also use Likert-scale annotation; 53 individuals registered, 15 teams submitted, and 9 system papers resulted.Source: §1 · p. 2

2 The LeWiDi 3 Benchmark

LeWiDi-2025 brings together four datasets covering sarcasm, irony, NLI, and paraphrase detection. They include varied annotation designs, and the released datasets share a harmonized format.

2 The LeWiDi 3 Benchmark

  • All four datasets are publicly available in a harmonized JSON format with common fields and dataset-specific information.Annotator age and gender are available for all four datasets, with additional attributes in some; metadata is distributed separately.Source: §2 · p. 2

2.1 The Conversational Sarcasm Corpus (CSC)

  • CSC contains around 7,000 English context–response pairs rated for sarcasm on a 1–6 scale by response generators and multiple external observers.Each speaker has 4–6 external observers; contexts describe imagined-interlocutor situations, and responses come from online participants.Source: §2.1 · p. 2

2.2 The MultiPICo dataset (MP)

  • MP is a multilingual perspectivist corpus of Twitter and Reddit post–reply pairs labeled for irony, with sociodemographic information about annotators.It covers nine languages, and each pair is typically annotated by an average of five workers.Source: §2.2 · pp. 2–3

2.3 The VariErr NLI dataset (VEN)

  • VEN was designed to distinguish annotation errors from legitimate human label variation in NLI; LeWiDi-2025 uses its first-round labels and explanations.Round 1 has 1,933 explanations for 500 re-annotated MNLI items; Round 2 contains 7,732 validity judgments.Source: §2.3 · p. 3

2.4 The Paraphrase Detection dataset (Par)

  • Par contains 500 QQP question pairs, each independently rated by the same four annotators on a -5 to 5 Likert scale for degree of paraphrase.The ratings include short textual explanations, and the dataset was new to LeWiDi-2025 participants.Source: §2.4 · p. 3

3 Task definition

The task provides a unified framework for learning from disagreement across heterogeneous datasets, encouraging shared approaches while allowing dataset-specific ones. Participants predict either population-level label distributions or individual annotators’ labels, using a three-phase competition and minimal baselines.

3 Task definition

  • Participants were encouraged to use a unified crowd-learning methodology across datasets, though dataset-specific approaches were allowed.The task’s stated goal is a unified testing framework for learning from disagreement and evaluating models on such datasets.Source: §3 · p. 3

3.1 Task A and Task B

  • Task A predicts a probability distribution over labels, while Task B predicts how a specific annotator would label an instance.Task A is evaluated against gold soft-label distributions; Task B measures agreement with actual annotator-level responses.Source: §3.1 · p. 3

3.2 Phases

  • The competition ran practice, evaluation, and post-campaign phases, moving from development-data testing to unseen test predictions and later public release of test data and labels.Missing evaluation submissions were replaced with organizer baseline scores.Source: §3.2 · p. 4

3.3 Baselines

  • The organizers provided random and most-frequent baselines, intentionally keeping them minimal to avoid discouraging participation.The most-frequent baseline assigns the training-set’s most frequent distribution or label, depending on the task.Source: §3.3 · p. 4

4 Evaluation metrics

LeWiDi-2025 evaluates both population-level soft-label predictions and individual annotator choices, using metrics suited to multiclass, multilabel, and ordinal settings.

4.1 Soft-label Evaluation

  • Soft-label evaluation compares predicted distributions with human-derived distributions, using Manhattan distance for MP and VEN4 and Wasserstein distance for ordinal Par and CSC.These distances replace cross-entropy, whose counterintuitive properties and multiclass comparison limitations are noted.Source: §4.1 · pp. 4–5
  • For Average Wasserstein Distance, moving probability mass between bins costs the absolute difference between their positions.This defines a symmetric, non-negative ground-distance matrix with zeros on the diagonal.Source: §4.1 · p. 5

4.2 Perspectivist Evaluation

  • Perspectivist evaluation models individual annotator choices, using error rate for nominal MP and VEN categories and normalized absolute distance for ordinal Par and CSC.Average error rate compares corresponding target and predicted vectors across samples; Average Normalized Absolute Distance averages normalized distances across samples.Source: §4.2 · p. 5

5 Participating systems

LeWiDi-2025 drew 53 registrants, 15 submitting teams, and 9 system papers. The systems ranged from annotator-aware transformers to LLM-based approaches and test-time scaling methods.

5 Participating systems

  • The task drew 53 Codabench subscribers and 15 teams submitting predictions, producing 9 system papers; Task A attracted broader participation than Task B.Six teams participated across all datasets and both tasks, while 11 teams engaged with Task B.Source: §5 · p. 5

5.1 Systems overview

  • Opt-ICL fine-tuned after post-training on over 40 disagreement-rich datasets, then predicted per-rater labels using annotator examples and demographics in prompts.It derived soft-label distributions from perspectivist predictions.Source: §5.1 · p. 5
  • DeMeVa selected ten demonstrations per annotator and found label-based selection better for multilabel datasets; PromotionGo ranked first on MP Task A.PromotionGo used augmentation, L1 loss aligned to Manhattan Distance, and an ensemble; DeMeVa derived soft-label distributions from perspectivist predictions.Source: §5.1 · pp. 5–6
  • McMaster found that nationality and ethnicity features yielded the largest performance gains among demographic features, while noting limits to relying on them.Its demographic-aware RoBERTa system was evaluated on all four datasets.Source: §5.1 · p. 6
  • Model Averaging and Majority Voting reliably boosted performance, while Best-of-N sampling did not transfer well from mathematical domains.The BoN Appetite Team investigated these three test-time scaling methods.Source: §5.1 · p. 6
  • Uncertain Mis(Takes) ranked first on VEN Task A by combining semantic entropy from clustered LLM explanations with text embeddings to predict soft-label distributions.The system generated 128 explanations per item and clustered them with a fine-tuned entailment model.Source: §5.1 · p. 6
  • NLP-ResTEAM combined soft-label and annotator-specific hard-label outputs, while LPI-RIT extended DisCo and twinhter trained separate text–annotator examples.NLP-ResTEAM used input tokens for annotator ID, demographics, behavior, or combinations; LPI-RIT added metadata and metric-aligned losses, while twinhter incorporated background information when available.Source: §5.1 · p. 6

6 Results and discussion

The results compare systems across two disagreement-prediction tasks, examine dataset-specific patterns, and assess new metrics and participant ablations. Performance patterns varied across datasets and evaluation metrics.

6.1 Results and statistics

  • OCP-ICL and DeMeVa ranked first and second overall; twinhter outperformed LLMs on Par and VEN, while tailored teams won locally on Par and VEN Task A.Ranks account for statistical ties identified with instance-level Wilcoxon signed-rank tests.Source: §6.1 · p. 6, §6.2 · pp. 7–8, Table 3 · p. 7

6.2 General discussion

  • Task A and Task B rankings were largely similar; all top systems used annotator-level predictions to derive soft labels, while teams pursuing multiple datasets generally used unified pipelines.Most used the same pipeline with a separate model per dataset; others used one model across datasets, and six teams extensively used annotator information.Source: §6.2 · pp. 7–8

6.3 Individual datasets results

  • On CSC, Opt-ICL found no significant gain from demographics, while dataset-specific fine-tuning was crucial; DeMeVa and McMaster reported RoBERTa fine-tuning comparable to larger-model in-context learning.The authors suggest CSC’s limited, partly missing age and gender data may explain the demographic result.Source: §6.3 · p. 8
  • On multilingual MP, most teams used multilingual backbones or fine-tuning, but the top two systems used no demographic data or saw only slight gains; only the Task B winner significantly beat baseline.The authors hypothesize the Task B pattern may relate to MP’s large number of annotators.Source: §6.3 · p. 8
  • On VEN and Par, perspectivist rankings were highly similar, and explanations—provided by annotators or generated by systems—contributed to stronger performance across both evaluation tasks.Twinhter ranked first, with Opt-ICL and DeMeVa tied second; Uncertain (Mis)Takes won VEN Task A using LLM-generated explanations and semantic entropy.Source: §6.3 · p. 8

6.4 The new evaluation metrics: an assessment

  • Manhattan and Wasserstein distances assess soft labels, while Error Rate and Average Normalized Absolute Distance assess annotator-level predictions; multilabel and ordinal metrics account for label structure and graded errors.Some Wasserstein upper bounds depend on the dataset, preventing direct cross-dataset comparisons; all metrics use zero for a perfect match.Source: §6.4 · p. 9

6.5 Post-Submission Experiments and Ablation studies

  • Post-submission studies found annotator-information effects varied by model and dataset: in-context rater examples helped LLMs, while metadata or embeddings were more influential for fine-tuned transformers.Augmentation effects also varied: OCP-ICL found gains only on MP, NLP-ResTeam reported benefits on Par and VEN, and PromotionGo found combined strategies worked best; teams used post-training, paraphrasing, back-translation, and lexical or translation-based augmentation.Source: §6.2 · p. 8, §6.5 · p. 9

7 Conclusions

The third LeWiDi edition drew active participation from researchers studying disagreement and variation. The authors identify open questions about ordinal-judgment evaluation and modeling annotators, and keep Codabench open for further submissions.

7 Conclusions

  • Participants actively explored how to use annotator information and how soft-label modeling relates to perspectivist modeling.Source: §7 · p. 9
  • Further work is needed on appropriate evaluation for ordinal judgments and on the usefulness of modeling individual annotators or annotator groups.Source: §7 · pp. 9–10
  • The authors intend the shared task and released datasets to stimulate further research, and Codabench will remain open to submissions after the deadline.Source: §7 · pp. 9–10

Limitations

The evaluation remained limited to text and did not directly test generalization to annotators unseen during training and development.

Limitations

  • The task remained restricted to text, leaving vision, speech, and multimodal disagreement-aware performance open.Source: Limitations · p. 10
  • Test-set annotators had also appeared in training and development, so generalization to unseen annotators was not directly evaluated.Source: Limitations · p. 10

A Evaluation Metrics for the Multilabel setting

The section adapts evaluation metrics for multilabel classification and describes MAMD as an average of label-specific distribution distances across samples.

A Evaluation Metrics for the Multilabel setting

A.1 Multilabel Average Manhattan Distance (MAMD)

  • MAMD averages Manhattan distances across label-specific distributions for each sample, then averages those values over all samples.Source: §A.1 · p. 13
  • For MAMD, N is the sample count, L the label count, n each distribution's length, and t and p the target and predicted values.Source: §A.1 · p. 13

A.2 Multilabel Error Rate (MER)

MER measures dissimilarity between predicted and target label vectors by averaging sample-level error rates.

A.2 Multilabel Error Rate (MER)

  • MER quantifies average dissimilarity between predicted and target label vectors across samples.Source: §A.2 · p. 13
  • For MER, N is the sample count, L the number of labels, a the annotator count per vector, and t and p the target and predicted vector elements.Source: §A.2 · p. 13

B Datasets specific leaderboards

This section presents dataset-specific results tables for CSC, MP, Par, and VEN.

B Datasets specific leaderboards

Loading 2510.08460v2…