Source-linked AI summary
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
Lifu Tu, Garima Lalwani, Spandana Gella, He He
TL;DR
The paper investigates why pre-trained language models can be robust to spurious correlations and how robustness changes when counterexamples are scarce. It analyzes pre-training and MTL on NLI and paraphrase identification, finding that both improve challenging-data performance mainly through generalization from minority examples. The authors conclude that robustness remains limited by training-data coverage because models do not extrapolate to unseen patterns.
Problem
The paper asks how pre-trained models generalize beyond spurious correlations and why their robustness gains are inconsistent when counterexamples are scarce.
Method
The study compares pre-trained and randomly initialized models and evaluates MTL with target and auxiliary datasets across NLI and paraphrase identification.
Results
Pre-training and MTL improve robustness on challenging datasets without hurting in-distribution performance, especially when target-dataset minority examples are scarce.
Takeaways & Limitations
Robustness depends on coverage of diverse patterns and generalization from minority examples rather than extrapolation to unseen patterns.
Takeaways & Limitations
Improvement remains limited by training-data coverage because current models do not extrapolate to unseen patterns.
Abstract
from arXiv · showhide
Recent work has shown that pre-trained language models such as BERT improve robustness to spurious correlations in the dataset. Intrigued by these results, we find that the key to their success is generalization from a small amount of counterexamples where the spurious correlations do not hold. When such minority examples are scarce, pre-trained models perform as poorly as models trained from scratch. In the case of extreme minority, we propose to use multi-task learning (MTL) to improve generalization. Our experiments on natural language inference and paraphrase identification show that MTL with the right auxiliary tasks significantly improves performance on challenging examples without hurting the in-distribution performance. Further, we show that the gain from MTL mainly comes from improved generalization from the minority examples. Our results highlight the importance of data diversity for overcoming spurious correlations.
1 Introduction
Robust NLP models must handle linguistic diversity beyond limited training distributions, where spurious correlations can produce high in-distribution but poor challenging-dataset performance. The paper argues that pre-training and MTL improve robustness through minority examples and underscores data diversity.
- Limited linguistic variation in training data creates a gap with real-world language diversity, encouraging models to rely on spurious correlations.
- High word overlap between premise and hypothesis can be associated with entailment, but this heuristic fails on challenging datasets where the correlation does not hold.
- Pre-training improves robust accuracy without using prior knowledge of the dataset’s spurious correlations.
- MTL improves robust accuracy in extreme-minority settings by improving generalization from minority examples.
- The results highlight data diversity as important for overcoming spurious correlations.
2 Challenging Datasets
The study evaluates robustness using challenging datasets that alter or counter spurious correlations present in standard NLI and paraphrase-identification benchmarks. It compares performance on matched in-distribution and challenging data.
- Challenging examples come from a different distribution where known spurious correlations do not hold, unlike held-out i.i.d. test examples.
- The analysis focuses on natural language inference and paraphrase identification, both supported by large-scale benchmarks with around 400k examples.
- NLI: HANS tests whether NLI models incorrectly treat high premise–hypothesis overlap as entailment by generating high-overlap non-entailment examples.
- PI: PAWS contains sentence pairs with high word overlap but different meanings, created through word swapping and back-translation.
- Table 2 compares MNLI-matched and QQP dev performance with HANS and PAWSQQP challenging-dataset performance.
3 Pre-training Improve Robust Accuracy
The paper examines how pre-training affects robustness across model sizes, pre-training-data scales, and training duration. Pre-trained models generally improve challenging-data performance, but gains vary by dataset and continue with longer fine-tuning.
- Models: Pre-trained models of different sizes and pre-training-data scales are compared with randomly initialized BERTBASE.
- Observations and inquiries: Pre-trained transformers improve challenging-dataset performance, but gains are less promising on PAWSQQP than on HANS.
- Observations and inquiries: Larger models and more training data yield a further improvement of 5 to 10 accuracy points on HANS, while improvement on PAWSQQP is marginal.
- Observations and inquiries: After three epochs, in-distribution accuracy saturates while challenging-dataset performance keeps increasing until around epoch 10, with more than 30% improvement.
3. What role does longer fine-tuning play?
The paper connects its robustness findings to a small amount of counterexamples in training data. These minority examples are presented as the basis for understanding how models generalize to challenging data.
- The paper’s empirical answers to its robustness questions are related to a small amount of counterexamples in the training data.
4 Generalization from Minority Examples
Pre-trained models improve robustness by generalizing from minority examples that counter spurious correlations, but different minority patterns require different amounts of data and training.
- 4 Generalization from Minority Examples: Pre-training improves out-of-distribution generalization from minority patterns rather than enabling extrapolation to entirely unseen patterns.
- 4 Generalization from Minority Examples: MNLI contains 727 HANS-like non-entailment examples and QQP contains 247 PAWS-like non-paraphrase examples that counter their respective spurious correlations.
- 4.1 Pre-training Improves Robustness to Data Imbalance: Removing minority counterexamples significantly reduces HANS accuracy, whereas removing equal numbers of random training examples preserves the original accuracy.
- 4.1 Pre-training Improves Robustness to Data Imbalance: After 6.4% of counterexamples are removed, most pretrained models approach random performance and become as poor as non-pretrained models; RoBERTaLARGE is slightly more robust.
- 4.2 Minority Patterns Require Varying Amounts of Training Data: HANS accuracy reaches 100% rapidly, while PAWS accuracy rises slowly and struggles to reach around 90% even with the full training set.
- 4.2 Minority Patterns Require Varying Amounts of Training Data: PAWS examples are longer and syntactically more complex than HANS examples, and accuracy decreases as sentence length and parse-tree height increase.PAWS averages 20.7 words and parse-tree height 11.4, compared with 9.2 words and 7.5 height for HANS.
- 4.3 Minority Examples Require Longer fine-tuning: Minority examples learn more slowly: their loss takes over 15 epochs to approach zero, and their dev accuracy plateaus around epoch 10 rather than around epoch 5 overall.
- 4.3 Minority Examples Require Longer fine-tuning: Longer fine-tuning improves minority-group performance without improving in-distribution accuracy, making i.i.d. dev-set early stopping insufficient for robustness.
5 Improve Generalization through Multi-task Learning
The paper uses multi-task learning to improve generalization from scarce minority examples by jointly training on target and auxiliary datasets. MTL improves challenging-dataset accuracy, especially for PAWS, while preserving in-distribution performance.
- 5.1 Multi-task Learning: MTL jointly trains on a target dataset and auxiliary datasets from the same or related tasks.The approach shares the pre-trained representation while using separate classification layers for each dataset.
- 5.1 Multi-task Learning: Related benchmark and challenging datasets are selected as auxiliaries to transfer useful knowledge and counter task-specific spurious correlations.The experiments use NLI and paraphrase-identification datasets as related tasks.
- 5.2 Results: MTL increases accuracy on challenging datasets across tasks without hurting in-distribution performance, particularly when target-dataset minority examples are scarce.The strongest gains are reported for settings such as PAWS.
- 5.2 Results: On HANS, MTL significantly improves BERTBASE but not RoBERTaBASE; RoBERTaLARGE reaches 75.7 (2.1) versus 77.1 (1.6) for STL.The authors suggest RoBERTa may already generalize sufficiently from MNLI minority examples.
- 5.2 Results: MTL consistently yields large improvements on PAWS across pre-trained models, where QQP contains fewer minority examples resembling PAWS patterns.This supports MTL as a way to improve generalization when minority examples are scarce.
- 5.2 Results: Ablations indicate that MTL’s robust-accuracy gains mainly come from improving generalization from minority examples in the target dataset.Removing target minority examples makes MTL only marginally better than STL on PAWSQQP, whereas removing auxiliary minority examples resembles random removal.
6 Related Work
The related work spans pre-training, data augmentation, robust learning algorithms, and transfer learning as approaches to improving robustness to distribution shift and spurious correlations.
- Pre-training and robustness: Pre-training studies report improved robustness to label noise, class imbalance, out-of-distribution detection, and cross-domain question answering.This work examines why pre-trained models appear more robust and emphasizes minority examples in training data.
- Data augmentation: Data augmentation improves robustness by adding target-distribution, syntactically rich, counterfactual, or recombined examples.These approaches aim to expose models to relevant variation or reduce reliance on spurious features.
- Robust learning algorithms: Robust learning algorithms focus on harder examples that do not enable shortcuts, but typically assume prior knowledge of the dataset’s spurious correlations.The paper instead provides evidence that generic data can improve out-of-distribution generalization.
- Transfer learning: Transfer learning and domain adaptation learn from one distribution while targeting a different but related distribution.Data selection and reweighting are common techniques, and source examples resembling target data can support transfer.
7 Conclusion and Discussion
The study attributes pre-training’s robustness gains to generalization from minority examples that counter dominant spurious patterns, with further gains from scale and multi-task auxiliary data. However, improvements remain constrained by training-data coverage and may not generalize across distribution shifts.
- 7 Conclusion and Discussion: Pre-training improves robustness by helping models generalize from minority examples that counter dominant spurious patterns.More pre-training data, larger models, and auxiliary data through MTL further improve robustness when minority examples are scarce.
- 7 Conclusion and Discussion: The robustness-accuracy trade-off may be exceeded with more data, but improvement remains limited by training-data coverage because models do not extrapolate to unseen patterns.The paper identifies increased data diversity through crowdsourcing or human-in-the-loop augmentation as a future direction.
- 7 Conclusion and Discussion: The study leaves open why pre-trained models do not overfit to minority examples and how initialization affects optimization and generalization.The authors frame these questions as important for designing better pre-training methods for robust models.
- 7 Conclusion and Discussion: Differences between HANS and PAWS indicate that semi-manually constructed challenging data may cover only specific distribution shifts.More comprehensive evaluation is needed because results may not generalize to other shift types.