Source-linked AI summary
Learning from others' mistakes: Avoiding dataset biases without modeling them
Victor Sanh, Thomas Wolf, Yonatan Belinkov, Alexander M. Rush
TL;DR
NLP models can rely on dataset-specific surface correlations, while effective prior debiasing methods may require those biases to be identified explicitly. This paper uses a limited-capacity learner's errors in product-of-experts training, reporting improved out-of-distribution generalization without explicitly modeling biases, alongside a trade-off with in-distribution performance.
Problem
Models can exploit surface correlations rather than intended task features, and existing bias-removal approaches rely on knowing the dataset biases, which can be costly to identify.
Source: §1 · p. 1Method
The method trains a limited-capacity weak learner and uses its errors in a product-of-experts ensemble to train a main model without explicitly specifying the biases.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
The authors conclude that weak learners can automatically identify and mitigate dataset biases, with higher-capacity weak learners bringing higher out-of-distribution performance and a trade-off in-distribution.
Source: §6 · p. 9The reported bias analysis uses biases referenced in the literature and does not rule out other biases being detected by the weak learner.
Source: §4.1 · p. 5
Abstract
from arXiv · showhide
State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the intended underlying task. Previous work has demonstrated effective methods to circumvent these issues when knowledge of the bias is available. We consider cases where the bias issues may not be explicitly identified, and show a method for training models that learn to ignore these problematic correlations. Our approach relies on the observation that models with limited capacity primarily learn to exploit biases in the dataset. We can leverage the errors of such limited capacity models to train a more robust model in a product of experts, thus bypassing the need to hand-craft a biased model. We show the effectiveness of this method to retain improvements in out-of-distribution settings even if no particular bias is targeted by the biased model.
1 INTRODUCTION
NLP models can exploit surface cues that correlate with labels rather than features of the intended task, contributing to brittleness on out-of-distribution examples. Because identifying dataset biases can be costly and difficult, this work trains robust models without explicitly specifying them.
1 INTRODUCTION
- Surface cues that correlate with labels can inflate benchmark performance while leaving models brittle on out-of-distribution or adversarial examples.For MNLI, hypothesis-only models can perform strongly, and negation words are often associated with contradiction labels.Source: §1 · p. 1
- Explicitly identifying dataset biases can require costly investigation or private annotation details, while preventing potential biases during new data collection is challenging.Prior de-biasing approaches construct a biased model, such as a hypothesis-only model for NLI, to help the main model learn where that model is wrong.Source: §1 · p. 1
- The paper trains a limited-capacity weak learner to exploit dataset biases, then uses its errors in a product-of-experts ensemble to train a more robust model without specifying the biases.The approach is evaluated on synthetic bias, natural language inference, and extractive question answering settings.Source: §1 · p. 2
- The authors report that explicit bias knowledge is unnecessary for improved out-of-distribution generalization, with weak-learner design choices trading in-distribution performance against robustness.The paper also reports that weak learners rediscover previously identified dataset biases.Source: §1 · p. 2
2 RELATED WORK
Related work addresses dataset bias through challenge-set evaluation, training-time debiasing, and data augmentation. This paper situates its approach as requiring neither explicit bias modeling nor the human-in-the-loop setup common to some augmentation methods.
2 RELATED WORK
- Challenge datasets test whether models rely on surface correlations; standard models can lose performance on these sets.Examples include challenge sets proposed for language understanding evaluation [Jia & Liang, 2017; McCoy et al., 2019b].Source: §2 · p. 2
- Training-time methods include adversarial training, example forgetting, dynamic loss adjustment, and product-of-experts debiasing.Prior product-of-experts methods explicitly model biases, whereas this work uses a more general setup without explicit bias modeling.Source: §2 · p. 2
- Data augmentation reduces bias by modifying dataset distributions, often using human knowledge or dynamic data collection; this work is mostly orthogonal and alleviates the need for a human-in-the-loop setup.Examples include swapping male and female entities [Zhao et al., 2018] and dynamic collection and benchmarking.Source: §2 · p. 2
- Despite progress from large pretrained language models, out-of-distribution generalization remains a challenge in NLP.The paper aims for robustness without significantly compromising in-distribution performance.Source: §2 · p. 2
- A parallel method also learns from shallow-model mistakes without explicit bias modeling, but uses the robust model's architecture for its weak model rather than a limited-capacity learner.The authors compare learner-capacity trade-offs and the high-capacity weak-model few-shot regime.Source: §2 · p. 2
3 METHOD
The method trains a weak model, then freezes it and trains a main model using a product-of-experts ensemble so the main model learns from the weak model's errors. Its gradient behavior depends on the weak model's certainty and correctness, and the authors relate the approach to distillation while using gold labels to identify errors.
3.1 OVERVIEW
- The two-stage method first trains weak learner fW with cross-entropy, then trains main model fM through a product-of-experts ensemble to learn from weak-model errors.The approach makes no assumption about dataset biases or a particular weak-learner architecture, capacity, or pretraining.Source: §3.1 · p. 3
- After weak-learner training, fW is frozen and only fM receives gradient updates; the PoE cross-entropy is the cross-entropy of the ensemble prediction.This differs from Mahabadi et al. (2020), who train weak and main models jointly.Source: §3.1 · p. 3
3.2 ANALYSIS: THE ROBUST MODEL LEARNS FROM THE ERRORS OF THE WEAK LEARNER
- For a positive binary-classification example, the analysis uses scalar logits w and m, with the weak learner's frozen logit and entropy measuring its certainty.Different values of w induce different gradient updates to the main-model logit m.Source: §3.2 · p. 3
- When the weak model is certain but incorrect, the main model receives gradients even when it classifies the example correctly, encouraging it to compensate for the weak model's mistake.This corresponds to low w and low weak-model entropy.Source: §3.2 · p. 3
- When the weak model is uncertain, the product of experts reduces to the main model and its gradient matches the standard cross-entropy gradient.The analyzed case has w = 0 and high weak-model entropy.Source: §3.2 · p. 3
- When the weak model is certain and correct, the main model's gradients are cropped early, and an extremely high weak-model logit can leave the example with no gradient.This limits the average gradient contribution from such examples.Source: §3.2 · p. 3
- MNLI examples occupy all three certainty-and-correctness regimes, including a nonempty certain/incorrect group whose examples encourage the model not to rely on dataset biases.Figure 1 summarizes the three gradient regimes and their presence in a real dataset.Source: §3.2 · p. 3, Figure 1 · p. 4
3.3 CONNECTION TO DISTILLATION
- Unlike distillation's student-teacher setup that encourages mimicry, PoE training encourages the main model to learn orthogonal behavior when the weak model is wrong, using gold labels to recognize errors.The final objective linearly combines standard cross-entropy and PoE cross-entropy, termed PoE + CE.Source: §3.3 · p. 4
4 EXPERIMENTS
The experiments examine whether weak learners rediscover known dataset biases, then test whether their errors can help train models that generalize to biased and adversarial evaluation settings. The results show benefits across controlled NLI and out-of-distribution NLI and QA evaluations, with trade-offs that vary by setting.
4 EXPERIMENTS
- The experiments use BERT-base as the main model and, by default, a smaller TinyBERT weak learner fine-tuned on the same data.The weak learner has 4M parameters versus 110M for BERT-base; the experiments examine bias discovery, a synthetic setup, and out-of-distribution evaluation.Source: §4 · p. 4, §4.1 · p. 5
4.1 WEAK LEARNERS REDISCOVER PREVIOUSLY REPORTED DATASET BIASES
- Among 1,000 high-loss, high-certainty errors, the weak learner’s contradiction predictions often involve hypothesis negation, while its entailment predictions often involve high lexical overlap.The analysis is based on previously reported biases and does not rule out discovery of other biases.Source: Table 1 · p. 5, §4.1 · p. 5
4.3 ADVERSARIAL DATASETS: NLI AND QA
- +24% improvement on HANS heuristic-non-entailed examples over standard cross-entropy; the weak learner relies heavily on word overlap out of distribution.On in-distribution MNLI, the weak learner has medium performance.Source: Table 2 · p. 6, §4.3 · p. 6
4.2 SYNTHETIC EXPERIMENT: CHEATING FEATURE
- When at least 60% of training examples contain the cheating feature, PoE performance drops more slowly than cross-entropy performance.As the feature prevalence approaches 1, PoE performance also drops because few biased examples remain that the weak learner gets wrong.Source: Figure 2 · p. 6, §4.2 · p. 6
4.3 ADVERSARIAL DATASETS: NLI AND QA
- +3% on AddSent and +2% on AddOneSent with multi-loss optimization, which also closes the in-distribution gap and improves adversarial robustness.PoE alone improves AddSent by +1% while sacrificing some in-distribution performance; unlike the MNLI/HANS experiments, multi-loss training improves out-of-distribution performance here as well.Source: §4.3 · pp. 6–7, Table 3 · p. 7
- On MNLI’s hard test set, the method surpasses PoE trained with a hypothesis-only biased learner.The set is expected to be less biased than MNLI’s standard split, and its examples are selected so a hypothesis-only model cannot predict labels accurately.Source: Table 2 · p. 6, §4.3 · p. 7
5 ANALYSIS
Weak-learner capacity changes the balance between in-distribution and out-of-distribution performance, while PoE training reduces alignment with the weak learner’s mistakes. Stronger weak learners can raise out-of-distribution accuracy at a substantial in-distribution cost.
5.1 REDUCING BIAS: CORRELATION ANALYSIS
- PoE-only training reduces correlation between evaluation losses and the biased model compared with training that combines PoE cross-entropy and standard cross-entropy.Figure 3 reports Pearson correlations; the passage notes that adding standard cross-entropy creates a trade-off.Source: §5.1 · p. 7, Figure 3 · p. 8
5.2 HOW WEAK DO THE WEAK LEARNERS NEED TO BE?
- 97% accuracy on HANS’s heuristic-non-entailed set with the 41.4-million-parameter weak learner came alongside 28% on MNLI and 3% on HANS’s heuristic-entailed examples.Out-of-distribution performance rises as weak-model capacity increases to a point, while in-distribution performance first drops slightly and then more strongly.Source: §5.2 · p. 8, Figure 4 · p. 8
- As weak-model capacity grows, certain-and-correct examples become more common and certain-and-incorrect examples less common, changing the main model’s gradient magnitudes.The certain-and-incorrect group generally aligns with out-of-distribution samples and induces higher-magnitude updates that encourage generalization at an in-distribution cost.Source: §5.2 · p. 8
6 CONCLUSION
The paper uses limited-capacity weak learners in a modified product-of-experts setup to train models that generalize better to out-of-distribution examples without explicitly knowing or modeling dataset biases. It also identifies a trade-off between out-of-distribution gains and in-distribution performance as weak-learner capacity increases.
6 CONCLUSION
- Limited-capacity weak learners in a modified product-of-experts setup can improve out-of-distribution generalization without explicitly knowing or modeling dataset biases.The conclusion describes this as an effective method for training models robust to dataset biases.Source: §6 · p. 9
- Higher-capacity weak learners lead to higher out-of-distribution performance, with a trade-off in in-distribution performance.Source: §6 · p. 9
- The authors see approaches that automatically identify and mitigate dataset bias as potential tools for future bias-discovery and mitigation techniques.Source: §6 · p. 9
A.1 EXPERIMENTAL SETUP AND FINE-TUNING HYPER-PARAMETERS
The experiments use TinyBERT weak learners and BERT-base-uncased robust models, with shared training choices and task-specific batch sizes and warmup steps. All experiments ran on a single 16GB V100 using half-precision training.
A.1 EXPERIMENTAL SETUP AND FINE-TUNING HYPER-PARAMETERS
- All experiments used half-precision training on a single 16GB V100.Source: §A.1 · p. 12
- For NLI, TinyBERT and BERT-base-uncased were trained for 3 epochs at learning rate 3e−5, batch size 32, and 2000 warmup steps; the multi-loss weights were 0.3 standard CE and 1.0 PoE CE.HANS results were averaged across 6 runs with different seeds.Source: §A.1 · p. 12
- For SQuAD, the models were trained for 3 epochs at learning rate 3e−5, batch size 16, and 1500 warmup steps; multi-loss weights were 0.3 standard CE and 1.0 PoE CE.Source: §A.1 · p. 12
- For fact verification, the models were trained for 3 epochs at learning rate 2e−5, batch size 32, and 1000 warmup steps; multi-loss weights were 0.3 standard CE and 1.0 PoE CE.Results were averaged across 6 runs with different seeds.Source: §A.1 · p. 12
A.2 ADDITIONAL EXPERIMENT: FACT VERIFICATION
The fact-verification experiment evaluates on FEVER and a symmetric hard test designed to counter claim-level artifacts. The method shows strong improvements on that symmetric test.
A.2 ADDITIONAL EXPERIMENT: FACT VERIFICATION
- The symmetric FEVER evaluation was designed to balance detected claim artifacts so that relying solely on statistical cues would yield a random guess.The set is described as challenging, and bias-reliant models drop significantly on it.Source: §A.2 · p. 12
- Table 4 reports accuracies on the FEVER dev set and the symmetric hard test.Source: Table 4 · p. 12
- The method shows strong improvements on the symmetric FEVER test set.The supplied passage reports no specific accuracy values.Source: §A.2 · p. 12
A.3 SOME EXAMPLES OF DATASET BIASES DETECTED BY THE WEAK LEARNER
The section presents qualitative examples of dataset biases detected by weak learners. The table caption states that weak learners can detect previously reported biases without explicitly modeling them.
A.3 SOME EXAMPLES OF DATASET BIASES DETECTED BY THE WEAK LEARNER
- Table 5 presents qualitative examples of dataset biases detected by a fine-tuned weak learner.Its caption states that weak learners detect previously reported biases without explicitly modeling them.Source: §A.3 · p. 13, Table 5 · p. 13
A.4 DETAILED RESULTS ON HANS AND MNLI MISMATCHED
The appendix reports detailed HANS and MNLI mismatched results, evaluates transfer across NLI benchmarks, examines a multi-loss trade-off, and compares weak-learner groups with Data Maps regions. The reported transfer results mostly match or surpass previously reported generalization performance, with stated exceptions.
A.4 DETAILED RESULTS ON HANS AND MNLI MISMATCHED
- Tables 6 and 7 report HANS results by heuristic and accuracies on MNLI dev mismatched and HARD test mismatched sets.Table 6 averages six runs and reports standard deviations.Source: §A.4 · p. 13, Table 6 · p. 13, Table 7 · p. 13
A.5 NLI: TRANSFER EXPERIMENTS
- The transfer evaluation uses SNLI training and tests on six NLI benchmarks plus the hard SNLI test set, selecting models on each dataset’s development set.The hard SNLI set is described as one a hypothesis-only model cannot solve easily.Source: §A.5 · p. 14
- Without explicitly modeling dataset bias, the method matches or surpasses previously reported transfer generalization, except on GLUE diagnostic and SNLI hard test.Table 8 reports transfer accuracies for BERT-base trained on SNLI; asterisks mark reported results.Source: Table 8 · p. 14, §A.5 · p. 14
A.6 INFLUENCE OF MULTI-LOSS OBJECTIVE
- On SQuAD/Adversarial SQuAD, F1 development score rises from 85.43 at α = 0.1 to 88.14 at α = 1.9, while AddSent falls from 56.67 to 55.06.Figure 5 presents the multi-loss trade-off; on MNLI/HANS, increasing standard cross-entropy weight raises in-distribution performance while lowering out-of-distribution performance.Source: §A.6 · p. 14, Figure 5 · p. 15
- The best-performing setup linearly combines PoE cross-entropy at weight 1 with standard cross-entropy weighted by α, using fixed coefficients throughout training.The multi-loss objective is described as similar to Utama et al. (2020)’s annealing mechanism.Source: §A.6 · p. 14
A.7 CONNECTION TO Data Maps SWAYAMDIPTA ET AL. (2020)
- The certain/incorrect group resembles Data Maps’ ambiguous and hard-to-learn regions, whose examples are reported to matter for out-of-distribution generalization.The authors say the weak model encourages the robust model to pay closer attention to certain/incorrect examples during training.Source: §A.7 · pp. 14–15
- The weak learner’s certain/correct examples lie in Data Maps’ easy-to-learn region, certain/incorrect examples in hard-to-learn, and uncertain examples mostly in ambiguous.The comparison plots 10,000 selected examples per group; the reverse mapping also shows overlap between regions and groups.Source: §A.7 · p. 15, Figure 6 · p. 15