On Robustness and Transferability of Convolutional Neural Networks

• The introduction asks whether transfer performance characterizes OOD performance more finely than in-distribution accuracy, a relationship not yet systematically explored.Source: §1 · pp. 1–2
• The study jointly evaluates in-distribution accuracy, OOD generalization without adaptation, and low-data transfer performance with adaptation.Under dataset shift, models cannot adapt using target-environment data; transfer learning uses related-task data to improve performance on a target task.Source: §1 · pp. 1–2
• A first meta-analysis of OOD metrics and transfer benchmarks across models up to 900M parameters finds that scaling data and models disproportionately improves transfer and OOD performance, but only marginally improves ImageNet validation performance.Source: §1 · p. 2
• For OOD robustness, the effect of model scale overshadows training-set size, training regime, and testing resolution.Source: §1 · p. 2
• The novel dataset enables fine-grained analysis of object size, location, and orientation; models become less sensitive to these factors as dataset and model sizes increase.Source: §1 · p. 2

Robustness of image classification models

• Dataset shift occurs when training and test distributions differ; this work focuses mainly on covariate shifts, where label conditionals agree but input marginals differ.Robustness is typically measured as expected accuracy on the test distribution.Source: §2 · p. 2

Dataset shift types

• Robustness benchmarks vary in scope: IMAGENET-C/P apply 15 corruption types at five severities, while OBJECTNET and IMAGENET-A [30] introduce departures from simple covariate shift.OBJECTNET’s clutter can make labels difficult to define, and IMAGENET-A selects naturally occurring examples misclassified by ResNets, creating model-specific selection bias.Source: §2 · p. 2
• IMAGENET-VID-ROBUST and YOUTUBE-BB-ROBUST [54] assess natural video invariance with pm-k, which requires correct classification of an anchor frame and up to k neighbors on each side.The metric counts a prediction as correct only when all 2k + 1 frames are classified correctly.Source: §2 · p. 2

Transferability of image classification models

• Transfer learning initializes a model for a new task with weights trained on a related pre-training distribution, then trains it using samples from the new task.The pre-training task may be auxiliary or self-supervised [15] [22].Source: §2 · p. 2
• VTAB [72] benchmarks transfer across 19 datasets with 1,000 labeled samples each, grouped as natural, specialized, and structured, with overall and subgroup scores.The groups range from natural tasks most similar to ImageNet to structured tasks least similar to it.Source: §2 · p. 2

• The meta-analysis evaluated 39 models across 23 robustness metrics and 19 transfer tasks, including average and per-group transfer performance.Transfer performance used VTAB-1K; the reported accuracy was the median across three fine-tuning runs.Source: §3 · p. 3
• Robustness metrics add little discrimination beyond ImageNet accuracy; non-ImageNet and video metrics are most informative, but their complementary signal remains low.The analysis compared logistic classifiers using ImageNet accuracy alone with classifiers adding up to two other metrics.Source: §3 · pp. 3–4, Figure 4 · p. 4
• ρ = 0.73 describes the correlation between average robustness and transfer, while residual robustness has only a weak ρ = 0.12 relationship with transfer.The robustness-transfer correlation is not stronger than the correlation between transfer and ImageNet accuracy, so aggregate robustness adds little predictive signal beyond ImageNet performance.Source: §3 · pp. 3–4
• For Specialized and Structured transfer tasks, no robustness metric predicts transferability well, and raw ImageNet accuracy is the best predictor for structured tasks.The analysis finds that robustness metrics do not relate to these challenging transfer tasks better than raw ImageNet accuracy.Source: §3 · p. 4

Pre-training dataset size impact

• With ResNet-101x3 on IMAGENET-21K, scaling from 112k steps on 1M data points to 1.12M steps on 13M yields 20-30% relative OOD classification error reductions; gains are least pronounced on YOUTUBE-BB(-W).For IMAGENET-21K, 1.12M steps can produce lower accuracy than 457k steps unless the full dataset is used; with JFT, longer training often improves accuracy even on smaller subsets.Source: §4 · pp. 5–6

Model size impact

• Replacing ResNet-50 with ResNet-101x3 reduces classification error by 5–20%; at fixed training duration, more data helps, but longer IMAGENET-21K training can hurt unless using the full dataset.The longer-training penalty is much less pronounced with JFT; gains from larger models are also least pronounced on YOUTUBE-BB/YOUTUBE-BB-W.Source: §4 · p. 5

Testing resolution and OOD robustness

• On IMAGENET-A and OBJECTNET, selecting the test resolution often raises accuracy, and FixRes often adds substantial gains; central-crop fine-tuning can help OBJECTNET without higher resolution.The figure compares authors’ default preprocessing and resolution with the best tested resolution, with and without FixRes.Source: Figure 5 · p. 5, §4 · p. 6
• Resolution changes or FixRes often slightly help IMAGENET-V2, rarely change IMAGENET-C results, and substantially help small video-robustness models but not larger ones.BiT models usually perform best at their authors’ suggested resolution, except on OBJECTNET and IMAGENET-A; FixRes adds little because BiT already uses it.Source: §4 · p. 6

Synthetic dataset details

• SI-SCORE enables controlled variation of object class, size, location, and orientation by placing foreground objects on uncluttered backgrounds.The dataset contains 614 object instances across 62 classes and 867 backgrounds, for 611,608 images.Source: §5 · p. 6, Figure 6 · p. 7
• Evaluation subsets vary factors through cross-products of values for each object-background pair, with two backgrounds sampled per object.Size and rotation evaluations retain objects at least 95% in the image; location evaluation does not apply that filter, though at least 25% of each object remains visible.Source: §5 · pp. 6–7

Learned invariances as a function of scale

• More pre-training data makes ResNet-50 and ResNet-101x3 more invariant to object location, rotation, and size, while ResNet-101x3 is more robust overall.Average prediction accuracy becomes more uniform across these variations; for location, outer regions improve in relative accuracy as dataset size increases, especially for the larger model.Source: Figure 6 · p. 7, §5 · p. 7

Related work

• Robustness research spans corruptions and transformations, texture bias, and adversarial attacks, but this paper focuses on average-case natural perturbations rather than adversarial robustness.Early studies found degradation under blur, Gaussian noise, occlusion, and compression, with less impact from color distortions [12]; later work examined additional corruptions and transformations [53] [31] [62] [17].Source: §6 · p. 7
• Robustness interventions include augmentation, self-supervision, transductive fine-tuning, adversarial training, and large-scale auxiliary pre-training, while [61] also identifies model and training-set size as beneficial.Noisy Student [70] reports gains on several robustness datasets, and Big Transfer (BiT) [36] reports strong OBJECTNET performance; [26] collects evidence for hypotheses in the literature.Source: §6 · pp. 7–8
• Transfer learning uses supervised, weakly supervised, or unsupervised pre-training, and although it achieves state-of-the-art results on many datasets, its implications for final robustness metrics remain unclear.Recent benchmarks assess transfer across datasets and tasks with differing characteristics or disjoint pre-training data [65] [72].Source: §6 · pp. 7–8
• Prior synthetic datasets insert objects onto backgrounds for training or evaluation, but do not systematically vary object size, location, or orientation, or isolate image-level translation and rotation robustness.The cited work includes synthetic-data training [75] [16] and evaluation [36].Source: §6 · p. 8

7. Limitations and future work

• Model and data scale together with a simple training recipe yield large improvements, yet models retain substantial performance gaps on OOD data.The authors conclude that further research is required.Source: §7 · p. 8
• Curated datasets and significant computing capabilities limit the practicality of this approach, making transfer learning—train once, apply many times—the most promising short-term paradigm for OOD robustness.Source: §7 · p. 8
• The study covers image classifiers fine-tuned to the IMAGENET label space, and models may share failure modes on datasets with IMAGENET-like biases [52].The authors note existing work showing that these models do not overfit to IMAGENET, but correlated failure modes remain possible.Source: §7 · p. 8
• The authors call for datasets enabling fine-grained analysis of important factors of variation and hope SI-SCORE will support that research.Source: §7 · p. 8
• SI-SCORE can probe qualitative model differences: ImageNet-trained ResNet-50 with GroupNorm does better on smaller objects, while BatchNorm does better on larger objects.A thorough investigation is beyond this work's scope; the contrast is shown in Figure 14b in the appendix.Source: §7 · p. 8
• Rather than requiring robustness to shifts, future work could ask whether models can quantify per-prediction uncertainty in substantially different deployment environments.This remains elusive for moderate-scale neural networks, but large-scale pretraining could potentially improve it.Source: §7 · p. 8

A. Analysis of existing robustness and transfer metrics

• Appendix A provides additional details on the analyses and benchmarks presented in Section 3.Source: A. Analysis of existing robustness and… · p. 12

A.1. Robustness metric correlation

• Figure 8 reports Spearman rank correlations between accuracies on eight robustness datasets across 39 models from various model families.The models are those presented in Table 2.Source: Figure 8 · p. 12

A.2. Dimensionality of the space of robustness metrics

• Only the first principal component of robustness-metric residuals was significantly above the null expectation.Residuals were computed after predicting each metric from IMAGENET accuracy; the null datasets independently permuted metric values to destroy their correlation structure.Source: §A.2 · p. 12
• To estimate dimensions beyond IMAGENET accuracy, each robustness metric was regressed on IMAGENET accuracy and its residuals were analyzed with principal components.The analysis considered the first four components and compared the residual variance explained with a null expectation from independently permuted metric values.Source: §A.2 · p. 12

A.3. Informativeness of robustness metrics

• OBJECTNET and YOUTUBE-BB were the most informative additions to IMAGENET accuracy for distinguishing model groups, although other combinations performed similarly within statistical uncertainty.The analysis compared classifiers using IMAGENET accuracy alone with classifiers adding one or two robustness metrics.Source: §A.3 · p. 12
• Logistic regression classifiers tested whether adding one or two robustness metrics to IMAGENET accuracy improved discrimination among the 12 model groups.IMAGENET accuracy alone served as the baseline, and results were expressed as accuracy differences from that baseline.Source: §A.3 · p. 12
• The metric-combination results were intended as a rough guide for practitioners with limited budgets, rather than a definitive ranking.Other combinations performed similarly to OBJECTNET and YOUTUBE-BB within statistical uncertainty.Source: §A.3 · p. 12

A.4. Visual Task Adaptation Benchmark Details

• The study evaluates on VTAB-1k, the 1000-example training-set version of the 19-task Visual Task Adaptation Benchmark.VTAB also permits use of the full datasets.Source: §A.4 · p. 12
• VTAB groups tasks as natural image classification, specialist-imaging domains, or scene classification requiring geometric understanding.Specialized images include medical images captured with specialist equipment.Source: §A.4 · p. 12
• The benchmark covers natural, specialized, and structured datasets, including image recognition, remote sensing, medical imaging, and geometric prediction tasks.Natural: Caltech101, CIFAR-100, DTD [9], Flowers102, Pets, Sun397, SVHN. Specialized: EuroSAT, Resisc45 [8], Patch Camelyon, Diabetic Retinopathy. Structured: counting and distance on CLEVR [33]; pixel-location and orientation on dSprites; camera elevation and object orientation on SmallNORB; object distance on DMLab [4]; vehicle distance on KITTI.Source: §A.4 · pp. 12–13

B. Scale and OOD generalization

• Models were pre-trained on ImageNet-21K and JFT, then fine-tuned on ImageNet for evaluation following the BiT-HyperRule setup [36].Source: B. Scale and OOD generalization · p. 13
• Pre-training varied dataset size from 1.28M to 13M examples and training schedules from 113K to 1.1M steps.These ranges span the tested combinations of dataset size and schedule.Source: B. Scale and OOD generalization · p. 14
• On JFT subsets, error reductions generally increase with training-set size and training duration, while scaling from ResNet-50 to ResNet-101x3 also increases relative error reduction.Figure 11 reports these trends for ResNet-50 and the relative reduction from ResNet-50 to ResNet-101x3.Source: Figure 11 · p. 14, B. Scale and OOD generalization · p. 14

C. Effect of the testing resolution

• Testing resizes each image so its shorter side is ⌊1.15 · r⌋, preserves aspect ratio, and takes a central crop of size r × r.At 224 × 224, this corresponds to standard single-crop preprocessing with the shorter side resized to 256.Source: C. Effect of the testing resolution · p. 14
• FixRes fine-tuning uses SGD with momentum 0.9 and initial learning rates of 0.004, or 0.0004 for BiT, scaled by batch size.The batch size is 2048 in most cases, and the learning rate is decayed by a factor of 10 after one-third and two-thirds of training iterations; the parameters were not extensively tuned.Source: C. Effect of the testing resolution · p. 14
• Figure 12 shows that higher evaluation resolution plus FixRes helps across many models and pretraining datasets; Figure 13 compares performance across resolutions before and after FixRes.The extended results cover all datasets and models.Source: C. Effect of the testing resolution · p. 14, Figure 12 · p. 15, Figure 13 · p. 16

D. Additional results on SI-SCORE, the synthetic dataset

• An ImageNet-trained ResNet-50 is more accurate on smaller objects with GroupNorm and on larger objects with BatchNorm.The paper highlights SI-SCORE’s potential for investigating such effects but leaves detailed investigation outside its scope.Source: Figure 14 · p. 16
• As dataset size increases, accuracy becomes more uniform across object areas, with stronger effects for larger models and objects covering 10–20% of pixels.The normalized-accuracy improvement is most pronounced for small object sizes.Source: Figure 15 · p. 17
• As dataset size increases, average prediction accuracy becomes more uniform across object rotations, with a stronger effect for larger models.Source: Figure 16 · p. 18
• As dataset size increases, sensitivity to object location decreases as outer regions improve in relative accuracy more than inner regions, especially for larger models.Source: Figure 17 · p. 19
• Location sensitivity also decreases when test images with objects less than 50% or 75% within the image are filtered, though the 75% effect is harder to see.The outer regions improve slightly more than the inner ones with the 50% filter; the 75% filter removes many edge pixels.Source: Figure 18 · p. 20, Figure 19 · p. 21

E. Overview of model abbreviations

• The model overview table defines SUP. as supervised pre-training and CH. as the width multiplier for the number of channels.Source: Table 2 · p. 22