Source-linked AI summary
On Robustness and Transferability of Convolutional Neural Networks
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D'Amour, Dan Moldovan, Sylvain Gelly, Neil Houlsby, Xiaohua Zhai, Mario Lucic
TL;DR
Modern image classifiers face distribution shifts, while transfer-learning results motivate studying whether scale and transfer performance relate to OOD robustness. The paper systematically analyzes these relationships across models, benchmarks, scaling factors, and preprocessing choices, finding that model and data scale improve transfer and OOD performance, while substantial OOD gaps and dataset-related scope limitations remain.
Problem
The paper addresses the limited systematic evidence about how in-distribution, OOD, and transfer performance relate across model and data scaling choices.
Method
The authors conduct a meta-analysis and scaling study across robustness and transfer benchmarks, varying model scale, training-set size, resolution, and preprocessing.
Results
Model and data scale disproportionately improve transfer and OOD performance while only marginally improving IMAGENET validation performance; preprocessing changes can also affect robustness.
Takeaways & Limitations
Scale and simple preprocessing choices are important considerations when evaluating and improving OOD robustness, although the studied models retain substantial performance gaps on OOD data.
Takeaways & Limitations
The study focuses on image classifiers fine-tuned to the IMAGENET label space, whose shared biases may produce correlated failure modes on related datasets.
Abstract
from arXiv · showhide
Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. However, several recent breakthroughs in transfer learning suggest that these networks can cope with severe distribution shifts and successfully adapt to new tasks from a few training examples. In this work we study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time and investigate the impact of the pre-training data size, the model scale, and the data preprocessing pipeline. We find that increasing both the training set and model sizes significantly improve the distributional shift robustness. Furthermore, we show that, perhaps surprisingly, simple changes in the preprocessing such as modifying the image resolution can significantly mitigate robustness issues in some cases. Finally, we outline the shortcomings of existing robustness evaluation datasets and introduce a synthetic dataset SI-Score we use for a systematic analysis across factors of variation common in visual data such as object size and position.
1. Introduction
The paper examines how in-distribution accuracy, OOD robustness, and transfer performance relate across model design choices. It systematically studies these relationships and finds that scale and preprocessing are important dimensions for robustness.
- Dataset shifts can substantially change deployment conditions, exposing image classifiers to sensitivity across photometric, natural, and adversarial perturbations.
- The paper asks whether transfer performance provides a more fine-grained characterization of OOD performance than in-distribution accuracy.
- The study systematically compares in-distribution accuracy, OOD generalization without adaptation, and low-data transfer performance with adaptation.
- The meta-analysis covers models ranging from self-supervised to fully supervised systems with up to 900M parameters.
- Increasing model and data scale disproportionately improves transfer and OOD performance while only marginally improving IMAGENET validation performance.
- The study analyzes robustness to object size, location, and orientation, finding lower sensitivity as dataset and model sizes increase.
2. Background
The background defines robustness as performance under dataset shifts and contrasts it with transfer learning, where target-task data can be used for adaptation. It reviews robustness and transfer benchmarks, including datasets designed to test varied shift types and task similarity.
- A dataset shift occurs when training and testing use different joint distributions, with this paper focusing mainly on covariate shifts where label conditionals remain unchanged.
- Robustness is typically measured by expected test accuracy under the shifted distribution.
- Dataset shift types: IMAGENET-V2, IMAGENET-C, IMAGENET-P, OBJECTNET, and video-derived datasets evaluate recollection, corruption, crowd-sourced, and naturally occurring shifts.
- Transferability of image classification models: VTAB contains 19 tasks with 1000 labeled samples each, grouped as natural, specialized, and structured according to their similarity to IMAGENET.
3. A meta-analysis of robustness and transferability metrics
The paper evaluates whether robustness metrics add information beyond ImageNet accuracy and how OOD robustness relates to transferability. Across 39 models, most robustness metrics add little discriminative or transfer-predictive information beyond ImageNet accuracy.
- Metric informativeness: Most tested robustness metrics provide little increase in model discriminability beyond IMAGENET accuracy.The analysis trains logistic classifiers using IMAGENET accuracy alone versus IMAGENET accuracy plus additional robustness metrics.
- Metric informativeness: Robustness metrics based on other datasets or video are more informative than IMAGENET-based metrics, but their additional discriminative power remains low.
- Robustness and transfer: Residual robustness correlates weakly with transfer performance, with ρ = 0.12 after accounting for IMAGENET accuracy.This indicates that aggregate robustness metrics add limited predictive signal for transferability beyond base IMAGENET performance.
- Robustness and transfer: Raw IMAGENET accuracy is the best predictor of transfer to structured tasks, while robustness metrics do not predict challenging transfer better.
- Robustness and transfer: Transfer performance and robustness correlate, but transfer exposes failures that are somewhat independent of robustness.
4. Scaling laws for OOD performance
The paper studies how pre-training data, model scale, training duration, and test-time preprocessing affect OOD performance. Larger data and models improve robustness, while resolution changes and FixRes can substantially improve performance under some distribution shifts.
- Experimental design: The study varies dataset size, ResNet architecture size, training schedule, and evaluation resolution to disentangle their effects on OOD robustness.It uses IMAGENET-21K and JFT pre-training, ResNet-50 and ResNet-101x3 models, and fine-tuning with the BiT HyperRule.
- Pre-training data size: Increasing pre-training data and duration produces 20–30% relative OOD error reductions for ResNet-101x3 on IMAGENET-21K.The comparison is between 112k steps on 1M data points and 1.12M steps on 13M data points; reductions are least pronounced for YOUTUBE-BB(-W).
- Model size: Increasing model size yields 5–20% reductions in classification error, while more data helps at fixed training duration.On IMAGENET-21K, training too long can increase error for larger models unless the full dataset is used, an effect less pronounced with JFT.
- Testing resolution: Selecting a higher testing resolution and applying FixRes often substantially improves IMAGENET-A and OBJECTNET accuracy.FixRes offers marginal benefits on IMAGENET but can produce substantial gains on robustness metrics; effects vary across datasets.
- Testing resolution: For IMAGENET-C, higher resolution usually adds little because the images are cropped to 224 pixels, whereas resolution adjustments address resolution mismatch in OOD evaluation.
- Synthetic evaluation: The synthetic dataset organizes evaluation around factors of variation and generates full cross-products when multiple factor values are specified.
5. SI-SCORE: A fine-grained analysis of robustness to common factors of variation
SI-SCORE is a scalable synthetic dataset for controlled evaluation of robustness to object size, location, and orientation. Using it, the authors find that larger pre-training datasets and models make predictions more uniform across these factors.
- SI-SCORE pastes segmented objects onto uncluttered backgrounds, enabling controlled variation of object class, size, location, and orientation.The dataset was designed as a scalable alternative to large-scale systematic data collection for fine-grained robustness analysis.
- 611 608 images combine 614 foreground objects from 62 classes with 867 backgrounds while varying object location, rotation angle, and size.
- As dataset size increases, sensitivity to object location decreases, with larger relative gains in outer regions; the effect is stronger for the larger model.
- Increasing pre-training data makes average prediction accuracy more uniform across object locations, sizes, and rotation angles.The larger ResNet-101x3 is also more robust than ResNet-50 in these analyses.
6. Related work
Prior work documents CNN sensitivity to natural and synthetic perturbations, while transfer learning studies emphasize the importance of model and dataset scale. Existing synthetic evaluation approaches had not systematically varied object size, location, and orientation.
- Image-classification robustness research has examined blur, noise, occlusion, compression, photometric changes, and geometric transformations.Adversarial robustness is related but distinct; this paper focuses on average-case robustness to natural perturbations.
- Data augmentation, auxiliary self-supervision, transductive fine-tuning, and adversarial training have been reported to improve selected robustness benchmarks.
- Earlier synthetic datasets inserted objects onto backgrounds but did not systematically vary object size, location, or orientation for robustness analysis.
7. Limitations and future work
The paper finds large improvements from jointly scaling models and data, but larger models still show substantial OOD performance gaps. The study is limited to ImageNet-label-space classifiers, and several broader questions remain open.
- Model and data scale together with a simple training recipe lead to large improvements in OOD generalization and transferability.
- The study considers image-classification models fine-tuned to the ImageNet label space and developed to optimize ImageNet test accuracy.The authors note that such models may have correlated failure modes on datasets sharing ImageNet biases.
- A thorough investigation of qualitative model differences using SI-SCORE is beyond this work’s scope.The authors suggest the dataset may support such future studies.
- Quantifying predictive uncertainty under deployment shifts remains an open direction that the authors leave for future work.
A.1. Robustness metric correlation
The appendix evaluates how robustness metrics relate to one another and how much they add beyond ImageNet accuracy. It uses correlations, residual variance, and model-group discrimination analyses across 39 models.
- Spearman’s rank correlation measures relationships among accuracies on eight robustness datasets across 39 models from multiple model families.
- Linear-regression residuals quantify robustness-metric variation not explained by ImageNet accuracy.
- Logistic-regression classifiers compare ImageNet accuracy alone with ImageNet accuracy augmented by one or two robustness metrics for discriminating 12 model groups.
A.4. Visual Task Adaptation Benchmark Details
VTAB-1k evaluates transfer using 19 tasks with 1000-example training sets, spanning natural, specialized, and structured visual problems. The robustness analysis uses multiple evaluation datasets and metrics to compare model behavior.
- VTAB-1k contains 19 transfer tasks, each using a 1000-example training set.
- The tasks cover natural image classification, specialized domains such as medical imaging, and structured geometric understanding.
- Robustness metrics are compared across model evaluations to assess how informative they are for distinguishing model types.
- The evaluation includes datasets spanning natural, remote-sensing, medical, and synthetic or structured visual tasks.
B. Scale and OOD generalization
The experiments vary pre-training data size, training duration, and model scale under a common pre-training and fine-tuning setup. Additional JFT-subset results examine how these factors affect classification error.
- Models are pre-trained on ImageNet-21K or JFT and fine-tuned on ImageNet using the BiT-HyperRule setup.
- Pre-training varies across five dataset sizes, from 1.28M to 13M examples, and five schedules, from 113K to 1.1M steps.
- JFT-subset experiments report classification-error reductions as functions of training steps and training-set size, including comparisons from ResNet-50 to ResNet-101x3.
C. Effect of the testing resolution
The evaluation-resolution analysis compares standard preprocessing with alternative resolutions and FixRes across models and datasets. It tests whether changing the evaluation pipeline improves performance without changing the underlying model.
- Evaluation uses a resize-then-central-crop procedure, with the standard 224 × 224 setting corresponding to resizing the shorter side to 256.
- FixRes fine-tunes models to target resolutions using a shared training strategy that was not extensively tuned for individual models or datasets.
- The extended evaluation compares default preprocessing, the best resolution, and FixRes across multiple resolutions, models, and datasets.
- The resolution comparisons include BiT variants trained with ImageNet or JFT pre-training and different ResNet scales.
- The BiT model labels distinguish the larger ResNet-101x3 variant from the ResNet-50x1 variant, with INet denoting ImageNet.
D. Additional results on SI-SCORE, the synthetic dataset
SI-SCORE provides synthetic evaluations across visual factors including object size, rotation, and position. Additional results examine how dataset size and normalization choices affect accuracy consistency across these factors.
- SI-SCORE: SI-SCORE shows higher accuracy for smaller objects with GroupNorm and for larger objects with BatchNorm in ImageNet-trained ResNet-50.
- Model comparisons: The study compares multiple model variants, including ResNet-50 and ResNet-101x3 models trained with ImageNet-21K or JFT.
- Object size: As dataset size increases, average prediction accuracy becomes more uniform across relative object areas.
- Rotation: As dataset size increases, average prediction accuracy becomes more uniform across object rotation angles.
- Object position: Location results normalize each grid position by the 95th-percentile accuracy to quantify spatial performance gaps.
- Object position: Additional location analyses filter out objects occupying less than 50% or 75% of the image.