Source-linked AI summary
What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
Yixin Nie, Xiang Zhou, Mohit Bansal
TL;DR
Most NLU benchmarks collapse subjective human judgements into majority labels, despite disagreement arising from context-dependent interpretations. The paper collects ChaosNLI’s large-scale opinion distributions and finds that models struggle on disagreement-heavy examples, motivating distribution-based evaluation.
Problem
Many pragmatic NLP tasks lack a single prescriptive ground-truth label because sentence meaning can vary with context or background knowledge.
Method
The paper collects 100 annotations per example for over 4k examples across SNLI, MNLI-matched, and αNLI to estimate collective human-opinion distributions.
Results
Models perform nearly perfectly on high-agreement examples but barely beat random guessing on low-agreement examples, where shared state-of-the-art errors concentrate.
Takeaways & Limitations
The paper argues that NLU models should be evaluated against collective human-opinion distributions, especially for descriptivist tasks such as NLI and commonsense reasoning.
Takeaways & Limitations
SNLI and MNLI examples were intentionally selected from development data with low human agreement, limiting their representativeness of those benchmarks overall.
Abstract
from arXiv · showhide
Despite the subjective nature of many NLP tasks, most NLU evaluations have focused on using the majority label with presumably high agreement as the ground truth. Less attention has been paid to the distribution of human opinions. We collect ChaosNLI, a dataset with a total of 464,500 annotations to study Collective HumAn OpinionS in oft-used NLI evaluation sets. This dataset is created by collecting 100 annotations per example for 3,113 examples in SNLI and MNLI and 1,532 examples in Abductive-NLI. Analysis reveals that: (1) high human disagreement exists in a noticeable amount of examples in these datasets; (2) the state-of-the-art models lack the ability to recover the distribution over human labels; (3) models achieve near-perfect accuracy on the subset of data with a high level of human agreement, whereas they can barely beat a random guess on the data with low levels of human agreement, which compose most of the common errors made by state-of-the-art models on the evaluation sets. This questions the validity of improving model performance on old metrics for the low-agreement part of evaluation datasets. Hence, we argue for a detailed examination of human agreement in future data collection efforts, and evaluating model outputs against the distribution over collective human opinions. The ChaosNLI dataset and experimental scripts are available at https://github.com/easonnie/ChaosNLI
1 Introduction
The paper challenges single majority labels as ground truth for pragmatic NLU tasks and introduces ChaosNLI to study collective human opinion distributions and model performance across agreement levels.
- Pragmatic NLP tasks can yield different interpretations because sentence meaning depends on context or background knowledge.
- ChaosNLI adds 100 annotations for over 4k SNLI, MNLI-matched, and αNLI examples, totaling 464,500 annotations.
- 10%, 20%, and 31% of collected αNLI, SNLI, and MNLI-matched examples, respectively, had original majority labels that failed to represent prevailing human opinion.
- State-of-the-art models fail to capture human opinion distributions, despite this ability differing from performance on the traditional accuracy metric.
- Models perform almost like solved systems on high-agreement examples but barely beat random guessing on low-agreement examples, where shared model mistakes concentrate.
- The paper argues for evaluating predictions against collective human-opinion distributions and examining agreement during future data collection.
2 Related Work
Related work documents reproducible disagreement in textual inference and alternative graded-label approaches, while positioning ChaosNLI as complementary collective-distribution analysis using standard NLI labels.
- Prior research found inherent, reproducible disagreement in textual inference using 50 independent annotators and roughly 19,840 graded annotations.
- ChaosNLI retains the original three-way NLI schema for direct comparison and complements graded-label work with collective-distribution analysis.
- Some studies replace categorical labels with finer-grained ordinal or real-valued schemas to address ambiguity.
3 Data Collection
ChaosNLI estimates human-opinion distributions by collecting 100 labels per example across three NLI dataset portions, using qualification, training, and performance monitoring to control annotation quality.
- ChaosNLI provides 100 annotations for each example in subsets of SNLI and MNLI-matched development sets and the entire αNLI development set.
- The three portions are named ChaosNLI-S, ChaosNLI-M, and ChaosNLI-α, corresponding to SNLI, MNLI-matched, and αNLI.
- Annotators label each context-statement pair as Entailment, Contradiction, or Neutral in a multi-round crowdsourcing interface.
- Quality control uses unanimous examples with over 95% majority agreement to test and track annotator performance.
- More than half of Turkers fail the onboarding test, while average accuracy on unanimous examples improves with experience and converges near 92%.
- The collection takes about one month over 464K annotations, with median annotation times ranging from 10 to 20 seconds per example.
4 Analysis of Human Judgements
Analysis shows that expanded annotation changes some benchmark majorities and reveals substantial variation in human uncertainty across ChaosNLI dataset portions.
- 10%, 25%, and 30% of ChaosNLI-α, ChaosNLI-S, and ChaosNLI-M examples, respectively, receive new majority labels different from the old labels.
- Collected labels provide finer-grained judgements, identify better prevailing majorities, and show multiple plausible interpretations for low-agreement examples.
- Human label distributions are approximated from 100 annotations per example, and entropy measures uncertainty or agreement among judgements.
- High entropy indicates low human agreement, whereas low entropy indicates high agreement.
- ChaosNLI-α has a roughly U-shaped entropy distribution, while ChaosNLI-S and ChaosNLI-M show single peaks, with MNLI-matched slightly skewed toward higher entropy.
5 Analysis of Model Predictions
ChaosNLI evaluates whether pretrained language models reproduce collective human-label distributions, rather than only majority labels, and how performance changes with human agreement. Models remain far from collective human performance, with distributional modeling diverging from accuracy and degrading sharply on disagreement-heavy examples.
- 5.2 Evaluation and Metrics: 100 annotations per example approximate human-label distributions, which are compared against neural-network softmax outputs.Jensen-Shannon Distance is the primary measure, with KL divergence as a complementary measure.
- 5.3 Main Results: Models’ JSD and KL scores remain substantially worse than estimated collective human performance across all three ChaosNLI sets.Collective human performance has both scores below 0.1, while the best model JSD exceeds 0.2 and the best KL barely falls below 0.5.
- 5.3 Main Results: Several models barely outperform or even underperform the chance baseline when predicting human-label distributions.On ChaosNLI-M, models have similar JSD to chance and worse KL; on ChaosNLI-α, BERT-base is worse on JSD and all models have much higher KL.
- 5.3 Main Results: Accuracy on majority labels does not visibly correlate with JSD or KL, indicating distinct abilities for majority prediction and distribution modeling.DistilBERT obtains the best KL on ChaosNLI-S and ChaosNLI-M despite the lowest majority-label accuracy there.
- 5.3 Main Results: Larger models are not consistently better at distributional metrics, although every large model improves accuracy over its base counterpart.On ChaosNLI-M, all large models have higher JSD than base models; large models also fail to beat base models on KL for ChaosNLI-α and ChaosNLI-M.
- 5.4 The Effect of Agreement: Model performance declines as human agreement decreases, with accuracy dropping from 0.9 to 0.5 and lowest-agreement bins approaching chance.JSD follows similar trends, and most common benchmark errors occur in low-agreement subsets where predictions are little better than random.
6 Discussion & Conclusion
The discussion questions majority-label evaluation for low-agreement examples and advocates evaluating models against collective human opinion distributions. Human agreement is also presented as an intrinsic property linked to modeling difficulty and calibration.
- Low human agreement characterizes most unsolved examples on current benchmarks, questioning the value of continuing majority-label evaluation.
- The paper recommends evaluating model outputs against collective human opinion distributions, especially for descriptivist tasks such as NLI and commonsense reasoning.
- Human agreement is an intrinsic property of each data point and can indicate modeling difficulty, suggesting connections to uncertainty estimation and calibration.
- Future work should explicitly model collective human opinions, study disagreement sources, and examine links between disagreement and language-understanding difficulty.
A Annotation Interface
The appendix shows the interfaces used to collect annotations for NLI and αNLI.
- Figure 6 shows the screenshot for NLI annotation collection.
- Figure 7 shows the screenshot for αNLI annotation collection.
- The two screenshots document the collection interfaces for NLI and αNLI, respectively.
B Hyperparameters
The appendix reports hyperparameter-selection procedures for SNLI, MNLI, αNLI, BERT, XLNet, and RoBERTa.
- SNLI and MNLI use the hyperparameters selected by their original authors.
- For αNLI, batch size, learning rate, and number of epochs were tuned.
- Only large BERT, XLNet, and RoBERTa models underwent parameter searches, while base models reused hyperparameters based on large-model results.
- Table 8 provides the hyperparameter details.
C Training Size and Trajectory
Training-trajectory plots for RoBERTa-large compare accuracy and JSD across four human-agreement bins as training data increases. Accuracy converges faster in high-agreement bins given sufficient training data.
- Figure 8 tracks RoBERTa-large accuracy and JSD across four human-agreement bins as training data increases logarithmically.
- The plots compare model training trajectories across bins defined by human-agreement levels.
- Accuracy converges faster on high-human-agreement bins given a fair amount of training data.
D Label Statistics
MNLI-m shows a shift in majority labels from neutral to entailment, which the authors tentatively associate with its multi-genre composition.
- MNLI-m shows a shift in majority labels from neutral to entailment.The authors suggest its multi-genre nature might explain the difference, while noting that cognitive or linguistic analysis remains future work.
E Other Details
The paper documents implementation resources, collection interfaces, training trajectories, performance curves, and hyperparameter settings through supporting figures and tables.
- The implementation used an Intel Xeon E5-2630 v4 CPU and four NVIDIA TITAN V GPUs.
- Table 6 lists URLs for downloaded external resources.
- Figures 6 and 7 show the interfaces used for NLI and αNLI collection.
- Figure 8 compares RoBERTa training trajectories and performance curves as training size increases on αNLI and SNLI@3+MNLI-m@3.
- Tables 7 and 8 report NLI label distributions and the best αNLI fine-tuning hyperparameters, respectively.