Source-linked AI summary
BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performance
R. Thomas McCoy, Junghyun Min, Tal Linzen
TL;DR
The paper asks whether repeated training of the same neural architecture produces similar linguistic generalizations. It fine-tunes 100 BERT instances on MNLI and evaluates them on MNLI and HANS, finding consistent in-distribution performance but wide variation in out-of-distribution generalization.
Problem
The paper asks whether linguistic generalization reflects a neural architecture or random differences between training runs.
Method
The authors fine-tuned 100 BERT instances on MNLI and evaluated them on the MNLI development set and HANS.
Results
MNLI development accuracy ranged from 83.6% to 84.8%, while out-of-distribution generalization varied dramatically across instances.
Takeaways & Limitations
Evaluating linguistic generalization requires multiple training runs and out-of-distribution tests, because similar in-distribution performance can conceal substantial differences in learned behavior.
Abstract
from arXiv · showhide
If the same neural network architecture is trained multiple times on the same dataset, will it make similar linguistic generalizations across runs? To study this question, we fine-tuned 100 instances of BERT on the Multi-genre Natural Language Inference (MNLI) dataset and evaluated them on the HANS dataset, which evaluates syntactic generalization in natural language inference. On the MNLI development set, the behavior of all instances was remarkably consistent, with accuracy ranging between 83.6% and 84.8%. In stark contrast, the same models varied widely in their generalization performance. For example, on the simple case of subject-object swap (e.g., determining that "the doctor visited the lawyer" does not entail "the lawyer visited the doctor"), accuracy ranged from 0.00% to 66.2%. Such variation is likely due to the presence of many local minima that are equally attractive to a low-bias learner such as a neural network; decreasing the variability may therefore require models with stronger inductive biases.
1 Introduction
The paper asks whether repeated training of one architecture yields consistent linguistic generalizations, distinguishing in-distribution from out-of-distribution generalization. Across 100 BERT fine-tuning runs, MNLI performance was consistent, whereas HANS performance varied substantially.
- In-distribution generalization concerns novel examples from the training distribution, whereas out-of-distribution generalization concerns examples from a different distribution.
- Standard NLP test sets usually assess in-distribution generalization, where Transformer models perform strongly.
- Strong in-distribution performance can reflect shallow heuristics rather than deeper linguistic knowledge, so standard tests may not reveal abstract language learning.
- The study examines whether linguistic generalization is consistent across instances of the same architecture, separating architectural effects from random initialization effects.
- The authors fine-tuned 100 BERT instances on MNLI and evaluated them on both MNLI development and HANS datasets.
- 83.6%–84.8% was the MNLI development accuracy range, while one HANS category ranged from 4% to 76% across the same instances.
2 Background
Prior work documented variability across training runs, including out-of-distribution variation, but this paper studies that variability in detail for BERT fine-tuned on natural-language data.
- Prior studies found that identical architectures can differ substantially in in-distribution generalization across training restarts.
- Earlier out-of-distribution studies used simple synthetic tasks, whereas this work tests variability after training on a natural-language dataset.
- The paper emphasizes depth over breadth by analyzing fine-grained categories within HANS rather than many models and datasets.
- The novel contribution is measuring variability across many BERT fine-tuning runs, whereas previous work usually used one or at most ten runs.
3 Method
The study uses natural language inference with MNLI for training and evaluates both matched-distribution performance and syntactic out-of-distribution generalization with HANS.
- Task and datasets: NLI presents a premise and hypothesis, requiring entailment, contradiction, or neutral as the output label.
- Task and datasets: The MNLI examples illustrate entailment, contradiction, and neutral labels for sentence pairs.
- Task and datasets: The MNLI matched development set measures in-distribution generalization because it was generated like the training set but withheld during training.
- Task and datasets: HANS measures out-of-distribution generalization using NLI examples designed to require syntactic understanding.
- Task and datasets: HANS includes cases where targeted heuristics succeed and cases where they fail, exposing models that apply a heuristic indiscriminately.
- Model and training: Each model combined BERT with a linear classifier and was fine-tuned on MNLI for three epochs.
4 Results
Across 100 BERT runs, in-distribution performance and example-level predictions were consistent, while HANS out-of-distribution performance varied substantially, especially on lexical-overlap cases requiring syntactic generalization.
- In-distribution generalization: 83.6% to 84.8%: all 100 BERT instances scored within this range on the MNLI development set.
- In-distribution generalization: 66% of MNLI development examples were answered correctly by all 100 instances, indicating substantial example-level agreement.
- In-distribution generalization: 93.1%: any pair of fine-tuned BERT instances agreed on this proportion of MNLI example labels across entailment, contradiction, and neutral.
- Out-of-distribution generalization: 5% to 55%: accuracy across runs varied on lexical-overlap examples inconsistent with the lexical-overlap heuristic, the most variable HANS category.
- Out-of-distribution generalization: 0% to 66%: subject-object swap accuracy varied across runs despite requiring only rudimentary subject-object distinctions.
- Out-of-distribution generalization: HANS evaluates subcases where structural heuristics either make correct entailment predictions or incorrect predictions requiring non-entailment labels.
5 Discussion
Models differing only in classifier initialization and training-example order can make substantially different out-of-distribution linguistic generalizations despite similar in-distribution behavior. The authors argue that this variability makes multiple restarts and out-of-distribution evaluation important, and may reflect many equally attractive local minima.
- Models differing only in initial weights and example order varied substantially in out-of-distribution linguistic generalization.The vast majority of initial weights, including all BERT weights, were held constant.
- Because similar in-distribution behavior can conceal substantial differences in learned generalizations, models should be evaluated across multiple restarts and on out-of-distribution data.The discussion links this recommendation to the observed cross-instance variation.
- Figure 6 partitions HANS performance by six heuristic-based categories and shows especially high variability for lexical-overlap cases inconsistent with the heuristic.The caption directs readers to Figure 7 for numerical results.
- Humans tend to converge on similar linguistic generalizations despite substantially different childhood linguistic input, unlike the models studied here.The authors suggest reducing model generalization variability could narrow a major gap in out-of-distribution generalization.
- Many equally attractive local minima may make the model’s selected solution sensitive to initialization and training-example order.The authors propose stronger inductive biases as one way to reduce this variability.