The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning

Anders Andreassen; Yasaman Bahri; Behnam Neyshabur; Rebecca Roelofs

Source-linked AI summary

The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning

Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, Rebecca Roelofs

arXiv:2106.15831v1Jun 30, 2021cs.LGcs.AIcs.CV

TL;DR

OOD accuracy usually follows a linear relationship with ID accuracy, making models that exceed this baseline rare. The paper empirically studies effective robustness throughout fine-tuning and finds that large-scale pre-training produces it during training, but it vanishes at convergence. Effective robustness rises with pre-training data size, diversity, and example difficulty, while high-accuracy maintenance strategies remain unresolved.

Problem
Models generally lose OOD accuracy relative to ID accuracy, and effective robustness beyond the established linear baseline is rare.
Method
The paper empirically tracks effective robustness during fine-tuning across pre-trained models, datasets, architectures, robustness benchmarks, and strategies for preserving it.
Results
Pre-trained models exhibit effective robustness during fine-tuning, with larger, more diverse, and more difficult pre-training data increasing it, but the effect vanishes at convergence.
Takeaways & Limitations
Effectively robust models make different predictions and correctly classify 10% of examples that no other current testbed model gets correct.
Takeaways & Limitations
Known approaches do not maintain high effective robustness in the high-accuracy regime.

Abstract

from arXiv · show

Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across a testbed of models. Models that are more accurate on the out-of-distribution data relative to this baseline exhibit "effective robustness" and are exceedingly rare. Identifying such models, and understanding their properties, is key to improving out-of-distribution performance. We conduct a thorough empirical investigation of effective robustness during fine-tuning and surprisingly find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We study how properties of the data influence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difficulty of the dataset. We also find that models that display effective robustness are able to correctly classify 10% of the examples that no other current testbed model gets correct. Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.

1 Introduction

Out-of-distribution performance usually follows a predictable linear relationship with in-distribution performance, making models that exceed this baseline rare and important to understand. This work studies effective robustness during fine-tuning and finds that pre-training produces it transiently, while convergence removes it.

1 Introduction: A linear ID–OOD accuracy trend explains part of OOD improvement through higher ID accuracy but leaves a persistent performance gap.This relationship has been observed across multiple robustness benchmarks and model types.
1 Introduction: Effective robustness measures OOD behavior that is qualitatively and quantitatively different from current standard testbed models.Understanding such models is presented as important for closing the ID–OOD gap.
1 Introduction: Models with high effective robustness are exceedingly rare, with only a handful among 204 evaluated ImageNet models showing nonzero effective robustness.The most notable example identified in prior work is zero-shot CLIP [29].
1 Introduction: Pre-trained models exhibit effective robustness during fine-tuning, whereas randomly initialized models do not.The study examines how effective robustness evolves rather than evaluating only converged models.
1 Introduction: The paper investigates how model size, dataset size, and example difficulty influence effective robustness and analyzes whether it can be maintained at high accuracy.Its contributions include studying prediction differences and reporting that effectively robust models correctly classify 10% of examples that no other current testbed model gets correct.

2 Related Work

Prior work establishes linear ID–OOD accuracy trends across shifted test sets and shows that effectively robust models are rare. It also motivates this paper by linking effective robustness to large-scale pre-training, zero-shot evaluation, and model similarity.

2 Related Work: Replication studies found positive linear relationships between original-test and shifted-test accuracy across CIFAR-10, ImageNet, MNIST, and SQuAD-style evaluations.The relationship has also been reported for object detection and natural and synthetic image shifts.
2 Related Work: Earlier ImageNet evaluations found positive-effective-robustness outliers primarily among models pre-trained on larger and more diverse datasets than ImageNet.Most models pre-trained on large datasets nevertheless showed no effective robustness.
2 Related Work: Zero-shot CLIP classifiers were reported to have larger effective robustness than CLIP models fine-tuned on downstream tasks [30].The present work compares this behavior with zero-shot evaluation of pre-trained ImageNet models.
2 Related Work: Theoretical work explains linear accuracy trends using model-similarity assumptions, while empirical work finds ImageNet prediction assignments more similar than chance.These findings suggest neural networks may learn a smaller function class than expected.

3 Effective Robustness

Effective robustness is defined relative to a fitted baseline linking ID and OOD accuracies across testbed models. The paper uses a logit-scaled linear fit and interprets effective robustness as the model’s OOD accuracy above that baseline.

3 Effective Robustness: Effective robustness is the difference between a model’s OOD accuracy and the baseline OOD accuracy predicted from its ID accuracy.The baseline is fitted from models evaluated on both fixed ID and OOD test sets.
3 Effective Robustness: The baseline is computed with a log-linear fit after applying the logit transformation to model accuracies.The authors report that logit scaling gives the best linear fit for both CIFAR-10 and ImageNet.
3 Effective Robustness: Figure 2 shows the testbed-model linear fit in rescaled logit-space and after transforming it back to linear-space.The fit establishes the reference relationship used to measure effective robustness.
3 Effective Robustness: Effective robustness is graphically the distance above the testbed line to a model’s OOD accuracy.The metric is distinct from absolute OOD accuracy: a model can have high effective robustness while retaining low absolute OOD accuracy.

4 Experimental setup

The experiments measure effective robustness throughout fine-tuning on ImageNet and CIFAR-10 using pre-trained and randomly initialized models across multiple architectures and natural distribution shifts. The setup varies pre-training datasets and evaluates against testbed-derived linear baselines.

4 Experimental setup: The study evaluates effective robustness throughout fine-tuning on ImageNet and CIFAR-10, two common downstream benchmarks for pre-trained models.CIFAR-10 enables transfer from widely available ImageNet pre-trained models.
4 Experimental setup: BiT models span ImageNet-1k, ImageNet-21k, and JFT pre-training, with additional ImageNet-pre-trained PyTorch architectures included.The BiT family uses ResNet-v2 architectures with Group Normalization and Weight Standardization.
4 Experimental setup: Robustness is evaluated on natural shifts including CIFAR-10.1, ImageNet-V2, ImageNet-R, and ObjectNet.The study focuses on naturally occurring shifts rather than synthetic image modifications.
4 Experimental setup: Figure 3 tracks effective robustness against ID accuracy at every fine-tuning epoch across architectures, pre-training datasets, and OOD benchmarks.It compares pre-trained trajectories with converged testbed models and, for selected tasks, randomly initialized models.
4 Experimental setup: Linear ID–OOD baselines use testbeds spanning low to high accuracy, drawing ImageNet results from and CIFAR-10 results from [32].These fits provide the reference needed to calculate effective robustness.

5 Results

Effective robustness emerges during fine-tuning in models pretrained on large, diverse datasets, but generally vanishes at convergence. Its peak increases with model and data scale, depends on fine-tuning example difficulty, and corresponds to predictions that recover examples missed by standard models.

5.1 Pre-Trained Models Have High Effective Robustness: Pretrained models exhibit high effective robustness during fine-tuning across robustness benchmarks and training configurations, whereas randomly initialized models do not.The finding holds across CIFAR-10.1, ImageNet-V2, ObjectNet, and ImageNet-R, as well as different architectures and fine-tuning choices.
5.1 Pre-Trained Models Have High Effective Robustness: At convergence, effective robustness vanishes on ImageNet-V2 but remains nonzero for ImageNet-R and ObjectNet, showing benchmark-dependent persistence.The same JFT-pretrained BiT model has identical ImageNet accuracy across these evaluations.
5.3 The Impact of Data on Effective Robustness: Maximum effective robustness increases with architecture size, pretraining-data size and diversity, and more difficult fine-tuning examples.Larger pretraining datasets eventually plateau, while harder examples increase robustness but reduce final in-distribution accuracy relative to random examples.
5.3 The Impact of Data on Effective Robustness: Switching from a larger pretraining dataset to a smaller one lowers final-task robustness by about 1% compared with using the larger dataset alone.BiT-M-1K receives extra ImageNet-1K fine-tuning after ImageNet-21K pretraining, whereas BiT-M-21K does not.
5.4 Analysis at Maximum Effective Robustness: At maximum robustness, pretrained models have shifted dominance probabilities and recover 10% of examples that no testbed model classifies correctly.The dominance pattern resembles zero-shot CLIP, while converged pretrained models align more closely with ordinary testbed models.

6 Conclusion

The paper identifies pre-trained models during fine-tuning as a broad class with effective robustness, while emphasizing that fine-tuning reduces this advantage and that high-accuracy ER remains unresolved.

6 Conclusion: Pre-trained models in the middle of fine-tuning and zero-shot pre-trained models exhibit high effective robustness across the studied settings.These models form an entire class of effectively robust models rather than isolated exceptions.
6 Conclusion: Effective robustness models make different predictions from standard models and correctly classify examples that no standard model gets right, supporting their value for model ensembling.The paper connects this prediction diversity to applications requiring complementary model behavior.
6 Conclusion: Effective robustness is lower after fine-tuning on the in-distribution task, leaving the construction of high-accuracy ER models as a future challenge.The paper presents its investigations as initial steps toward enhancing ER in the high-accuracy regime.

A Scaling for Linear Fit

The paper models ID–OOD accuracy with a linear fit after scaling accuracies, selecting logit scaling as the best fit for CIFAR-10 and ImageNet before describing the experimental model suite.

A Scaling for Linear Fit: Probit and logit scalings are considered because nonlinear accuracy transformations improve linear fitting, especially for very high- and low-performing models.Prior work used probit scaling, while the paper also evaluates logit scaling.
A Scaling for Linear Fit: Logit scaling gives the best linear fit for both CIFAR-10 and ImageNet and is used throughout the work.The fit is shown in logit-scaled space and transformed back to linear space for CIFAR-10.
A Scaling for Linear Fit: The fit transforms original accuracy with a logit-based affine relation before applying the inverse logit to obtain new accuracy.The authors compute numerical fit values for this functional form and list them in Table 3.
B.1 Pre-trained Image Classification Models: The experiments use diverse architectures and model sizes, including ImageNet-pre-trained models and Big Transfer models with varied pre-training datasets.BiT variants span ImageNet-1k, ImageNet-21k, and JFT pre-training, with additional JFT fractions and zero-shot CLIP considered.
B.2 Data Preprocessing: CIFAR-10 images are resized to 224x224 for both pre-trained and randomly initialized models, while ImageNet uses standard crop and normalization procedures.Fine-tuning and evaluation preprocessing differ: training uses augmentation, whereas test evaluation uses resizing, cropping, and normalization.

B.4 Confidence Intervals, Averaging Runs, and Error Bars

The appendix specifies uncertainty and averaging procedures for effective robustness, using five runs, 100-bin averages, and conservative maximum-standard-deviation reporting.

B.4 Confidence Intervals, Averaging Runs, and Error Bars: CIFAR-10 and CIFAR-10.1 testbed accuracies use 95% Clopper–Pearson confidence intervals.These intervals apply to the testbed models described in the main experimental setup.
B.4 Confidence Intervals, Averaging Runs, and Error Bars: Effective robustness measurements averaged across runs use a binned average over 100 bins, with standard deviation computed across five runs.Figure 6 illustrates the resulting per-bin variability for fine-tuned models.
B.4 Confidence Intervals, Averaging Runs, and Error Bars: Randomly initialized CIFAR models align with standard 32x32 testbed models, indicating that resizing to 224x224 does not affect the reported ER result.The authors also independently confirmed the same result using 32x32 images.
B.4 Confidence Intervals, Averaging Runs, and Error Bars: For maximum effective robustness, the authors report the maximum standard deviation across all bins because the maximum-ER bin can have run-dependent variation from unequal bin sizes.This is chosen as a conservative alternative to using the standard deviation of only the maximum-ER bin.
B.4 Confidence Intervals, Averaging Runs, and Error Bars: All models use five runs with stochastic gradient descent, momentum 0.9, and weight decay 10^-4 unless otherwise specified.Training settings vary by model family, including batch size, epoch count, learning rate, and whether only the prediction head is trained.
B.4 Confidence Intervals, Averaging Runs, and Error Bars: For a BiT-R101x3 model, random initialization shows no ER, whereas JFT pre-training produces ER during training that vanishes at convergence.This comparison is shown for both ImageNetV2 after ImageNet training and CIFAR-10.1 after CIFAR-10 training.

C.2 ImageNet Robustness Benchmarks

For a JFT-pre-trained BiT model fine-tuned on ImageNet, ER vanishes on ImageNetV2 at convergence but remains nonzero on ImageNet-R and ObjectNet.

C.2 ImageNet Robustness Benchmarks: At the end of fine-tuning, ImageNetV2 ER reaches zero while ImageNet-R and ObjectNet retain nonzero ER for the same pre-trained model.The model is a BiT-L-R152x4_JFT fine-tuned on ImageNet and evaluated on all three robustness benchmarks.
C.2 ImageNet Robustness Benchmarks: ObjectNet accuracy stops at around 60%, and the paper does not explain why this occurs for ObjectNet and ImageNet-R but not ImageNetV2.This remains an unresolved dataset-specific observation in the appendix.

C.3 Architecture Independence

Effective robustness during fine-tuning is consistently associated with pre-training rather than architecture, loss, learning rate, or which layers are trained. Pre-trained models show ER across architectures, whereas randomly initialized models do not, and zero-shot evaluations also exhibit ER.

Architecture and initialization: Pre-trained models exhibit ER across architectures, while randomly initialized counterparts show no significant ER.This pattern remains conceptually stable across learning rates.
Loss functions: ER evolution is remarkably similar under mean-square-error and cross-entropy training, despite a higher maximum for MSE in one experiment.The authors caution that the higher MSE maximum may not generalize.
Optimization: Changing the learning rate produces only small variations and preserves the distinction between pre-trained and randomly initialized models.The comparison uses a fixed Wide-ResNet-50-2 architecture.
Fine-tuning scope: ER during fine-tuning is largely independent of whether the full network or only the last layer is trained.Full-model training reaches higher final accuracy, but ER evolution is mostly similar.
Zero-shot evaluation: Most zero-shot models show some ER, but they are not significantly more robust than fine-tuned pre-trained models.ImageNet logits are combined by max, mean, or sum after mapping classes to CIFAR-10.

C.8 Dataset Size and Diversity

Effective robustness generally increases with pre-training dataset size, diversity, and model size, while difficult fine-tuning examples also raise ER at the cost of in-distribution accuracy. A notable small-model exception remains unexplained.

Dataset size and diversity: Maximum ER increases with pre-training dataset size for BiT-R101x3 models, although gains plateau beyond 32 million JFT images.The plateau may reflect a fixed number of pre-training iterations rather than dataset size itself.
Dataset size and diversity: Larger and more diverse pre-training datasets produce higher maximum ER during CIFAR-10 fine-tuning.The trend is observed across multiple BiT architecture sizes, with about a 1 percentage-point ER drop after intermediate ImageNet-1k fine-tuning.
Model size: Larger model architectures generally achieve higher maximum ER for fixed pre-training datasets, except for the anomalous BiT-S-R50x1 model.The exception exceeds comparable larger BiT-S architectures and other R50x1 models.
Example difficulty: Training on more difficult examples increases ER, but using only hard examples lowers final in-distribution accuracy.Difficulty is measured with the C-score while maintaining class balance.
Example difficulty: Gradually phasing out easy examples is the most consistent tested strategy for retaining high in-distribution accuracy while ending training on difficult examples.Switching directly to hard examples raises ER but also reduces in-distribution accuracy.

C.11 Replay Buffer: Replaying Images from the Pre-trained Dataset

Replay buffers preserve accuracy on the pre-training dataset but do not maintain effective robustness on the target dataset at high in-distribution accuracy. Class-mapped replay and weight regularization likewise fail to improve ER retention.

Replay buffer: Maintaining high accuracy on the pre-training dataset does not translate into higher ER on the closer target OOD dataset.Replay preserves ImageNet accuracy but fails to maintain high ER at high CIFAR-10 accuracy.
Replay buffer: Replay buffers do not prevent effective robustness from disappearing toward the end of CIFAR-10 fine-tuning.This occurs despite preserving substantially higher ImageNet accuracy during fine-tuning.
Class mapping: Using a shared class-mapped prediction head with equal or 100-times more ImageNet images does not improve ER retention.The experiment trains on CIFAR-10 and replayed ImageNet examples with aligned target classes.
Weight regularization: Small L2 regularization toward the pre-trained prediction-head weights does not improve ER, while large regularization prevents reaching high accuracy.The regularization introduces a trade-off without solving ER loss.

D Shape of the Effective Robustness Curve

During fine-tuning, ER typically rises from near zero to a peak and then declines, but part of this shape is induced by the logit-space fit and ER definition. Mixed classifiers can generate ER theoretically, without achieving both high ER and high in-distribution accuracy.

ER curve: Pre-trained models typically show ER that starts near zero, peaks during fine-tuning, and diminishes by convergence.The same qualitative curve is reported for CIFAR-10 and ImageNet.
ER curve: The ER of the y = x reference curve also peaks, showing that the apparent rise-and-fall is partly induced by the logit-space fit and ER definition.After accounting for this effect, the reference curves are mostly decreasing during fine-tuning.
Mixed classifier: Mixed classifiers obtain ER by randomly sampling predictions from low- and high-accuracy models, tracing a line that benefits from convexity in the logit-scaled fit.They are theoretically interesting but cannot combine high ER with high in-distribution accuracy in practice.