Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization

• Classical domain-adaptation bounds allow wide variation in out-of-distribution accuracy, whereas prior dataset reconstructions observed near-linear correlations.The paper asks how broadly those correlations apply beyond reconstructions and common convolutional architectures.Source: §1 · pp. 1, 3
• The trends hold across architectures, classical methods, hyperparameters, training-set sizes, and training durations.The evaluated methods include convolutional networks, visual transformers, self-supervised models, logistic regression, nearest neighbors, and kernel machines.Source: §1 · p. 3
• Less regular relationships occur for some CIFAR-10-C corruptions, Camelyon17-WILDS hospital shifts, and an alternative iWildCam-WILDS split.The paper studies these cases with additional experiments to identify possible causes of the missing or weaker linear trends.Source: §1 · pp. 3–4
• Pretraining sometimes preserves the same in-distribution/out-of-distribution relationship but can produce a distinct trend, especially on iWildCam-WILDS.ImageNet pretraining follows the same trends on CIFAR-10 and FMoW-WILDS, whereas iWildCam-WILDS shows clearly different relationships.Source: Figure 1 · p. 2, §1 · p. 4
• A Gaussian data model identifies distribution-shift covariance structure as a factor affecting performance correlation.The candidate theory correctly identifies this property for Gaussian-noise corruption on CIFAR-10-C.Source: §1 · p. 4
• Precise linear trends link in-distribution and out-of-distribution accuracy across many models and distribution shifts.The relationship appears across benchmark reproductions, pose estimation, satellite imagery, wildlife classification, and other image-classification shifts, despite potentially large accuracy gaps.Source: Figure 1 · p. 2, §1 · pp. 3–4

Evaluation distributions

• The setup defines in-distribution performance on held-out samples from training distribution D and out-of-distribution performance on a separate distribution D1.For loss ℓ, performance is represented as the expected loss over the corresponding distribution.Source: §2 · pp. 4–5

Experimental procedure

• The core procedure trains independent models on D, evaluates each on held-out D and shifted D1 test sets, then examines paired performances in scatter plots.Models vary in training set size, architecture, random seed, and optimization algorithm.Source: §2 · p. 5

Visualization

• Probit-scaled axes make bounded-loss trends substantially more precise, with each point representing one model and 95% Clopper-Pearson confidence intervals shown for accuracies.The probit transform maps [0,1] to (-∞,+∞).Source: §2 · p. 5

Unified testbed

• The unified testbed spans classical methods and convolutional neural networks, covering more than 3,000 trained models and 100,000 test-set evaluations.Detailed testbed specifications are deferred to Appendix A.Source: §2 · p. 5

3.1 Distribution shifts with linear trends

• Linear trends appear across five datasets and four broad shift categories, spanning dataset reproductions, benchmark changes, synthetic perturbations, and real-world WILDS shifts.The experiments cover CIFAR-10, ImageNet, FMoW-WILDS, iWildCam-WILDS, and YCB-Objects; Table 1 summarizes the observed ID–OOD pairs.Source: §3.1 · p. 5, Table 1 · p. 6
• Linear trends hold for dataset reproductions, compatible benchmark shifts, many synthetic perturbations, and real-world shifts, but exceptions include isotropic Gaussian noise.Deep and classical CIFAR-10 models follow trends on CIFAR-10.1, CIFAR-10.2, CINIC-10, and STL-10; YCB-Objects models do so under lighting and texture changes; FMoW-WILDS and iWildCam-WILDS models do so across temporal, subpopulation, and camera-trap shifts.Source: §3.1 · pp. 6–7

3.2 Variations in model hyperparameters, training duration, and training set size

• The same linear trend persists when varying model hyperparameters, training duration, or training-set size, with each change moving models along rather than altering the fit.On CIFAR-10 and CIFAR-10.2, the reported fits have R2 “ 0.99 across model-size, regularization, optimization, training-duration, and dataset-size variations; the same invariance also holds for CINIC-10, CIFAR-10-C Fog, YCB-Objects, and FMoW-WILDS.Source: §3.2 · p. 7

4.1 Camelyon17-WILDS

• Camelyon17-WILDS is a hospital-held-out tissue classification task whose shift mainly reflects staining and imaging differences across hospitals.ImageNet pretraining does not increase the ID–OOD correlation, whereas strong augmentation reduces but does not eliminate OOD variation.Source: §4.1 · p. 8
• Camelyon17-WILDS models show highly variable OOD accuracy despite similar ID accuracy, with no precise linear trend.At 95% ID accuracy, OOD accuracy ranges from about 50% to 95%, and this variability persists after averaging over ten runs.Source: §4.1 · p. 8, Figure 3 · p. 9

4.2 CIFAR-10-Corrupted

• CIFAR-10-C correlations depend strongly on corruption, while covariance-matched Gaussian noise yields a much better fit than isotropic noise.The reported fit improves from R2 = 0.44 with isotropic Gaussian noise to R2 = 0.93 when the OOD covariance matches the ID covariance.Source: §4.2 · p. 9, Figure 4 · p. 10
• Across CIFAR-10-C and ImageNet-C, each corruption exhibits a more linear trend on ImageNet-C than on CIFAR-10-C.The discrepancy remains unexplained and is identified as a direction for future work.Source: §4.2 · p. 9

Overview

• The section finds that task and pretrained-model use determine whether ID/OOD linear trends persist, rather than uniformly breaking down under auxiliary pretraining.Models pretrained on seemingly broader distributions can still follow the trend of models trained purely on ID data.Source: §5 · p. 10

Fine-tuning pretrained models on ID data

• Fine-tuning improves both ID and OOD performance, preserves the existing trend on CIFAR-10 and FMoW-WILDS, but changes it on iWildCam-WILDS and with weight decay.Figure 5 summarizes these relationships across CIFAR-10.2, FMoW-WILDS, and iWildCam-WILDS.Source: Figure 5 · p. 11, §5 · p. 11
• A possible explanation for iWildCam-WILDS is that ImageNet pretraining encodes viewpoint invariances useful for unseen camera viewpoints, unlike the overhead-view FMoW-WILDS setting.The paper presents this as a conjecture and suggests investigating invariances involving lighting, object pose, and background.Source: §5 · p. 11

Zero-shot prediction on pretrained models

• Zero-shot ImageNet models on CIFAR-10 deviate from the basic trend and follow a different trend closer to x = y, suggesting ID-training bias partly explains OOD performance drops.The remaining gap below x = y is attributed to CIFAR-10 reproductions being harder for current methods.Source: §5 · p. 12

Additional experiments

• Linear probing, task-relevant pretraining, and semi-supervised training generally deviate less from the basic trend than zero-shot prediction, with similar conclusions across additional OOD settings.These experiments include CIFAR-10.1 and alternative FMoW-WILDS region subsets.Source: §5 · p. 12

• Linear in-distribution–out-of-distribution trends hold across model families, architectures, hyperparameters, and training durations.Source: §6 · p. 12
• Probit or logit scaling on both axes makes the linear trends more precise.Source: §6 · p. 12
• Changing in-distribution training-data quantity does not affect the linear trend, whereas pre-training on another distribution can change it.Source: §6 · p. 13
• Distribution shifts differ in trend precision: some are precise, while others work only for subsets of models or not at all.Source: §6 · p. 13

7.1 A simple Gaussian distribution shift setting

• The model uses binary labels and class-conditional isotropic Gaussians, with shifts represented by changes in mean and variance: µ1 = αµ + β∆ and σ1 = γσ.The shift direction ∆ is uniformly distributed on the sphere and independent of classifiers trained only on the original data.Source: §7.1 · p. 13
• Theorem 1 shows that, when the random shift direction is independent of the classifier, probit-transformed ID and OOD accuracies are nearly linear with slope α/γ.The deviation from linearity is O(d^-1/2) and vanishes in high dimension.Source: §7.1 · p. 13
• Figure 6 shows logistic and ridge regression accuracies agree with Theorem 1’s prediction, while nearest neighbors and random forests exhibit the same linear relationship, despite this not following directly from theory.Figure 6 uses d = 10^5, α = 0.7, β = 0.5, and γ = 1.Source: Figure 6 · p. 14, §7.1 · p. 14

7.2 Modeling departures from the linear trend

• The linear trend can become highly non-linear when an adversary selects the shift using knowledge of tested models or when pretraining data reveals the shift direction.Pretrained models may lie above the linear fit of classifiers without pretraining.Source: §7.2 · p. 14
• Non-isotropic covariance can break linearity under additive Gaussian noise, whereas scalar covariance scaling preserves it; covariance-matched CIFAR-10 noise yields higher ID-OOD correlation than isotropic noise.These observations suggest covariance changes may explain departures such as CIFAR-10-C-Gaussian, while the precise distinction among real shifts remains open.Source: §7.2 · pp. 14–15

Domain generalization theory

• Domain-generalization bounds use classifier-induced divergence measures to relate source and target error, but the paper’s linear trends show predictable OOD performance that can remain significantly different from ID performance.The Gaussian model illustrates this phenomenon for particular distributions and linear classifiers, unlike the broader domain-generalization bounds.Source: §8 · p. 15
• Mania and Sra [69] proposed a model-similarity condition implying approximately linear ID–OOD accuracy trends, based on stable probabilities of certain multiple-model error events under shift.Whether this condition explains which shifts produce precise trends remains an open question.Source: §8 · p. 15

Empirical observations of linear trends

• Earlier dataset-reproduction studies found similar linear trends for CIFAR-10, ImageNet, MNIST, and SQuAD, while later work extended them to ImageNet-related shifts and alternative accuracy metrics.These studies were initially limited to dataset reproductions or ImageNet-like tasks.Source: §8 · p. 16
• The paper broadens linear-trend evidence to CINIC-10, STL-10, FMoW-WILDS, iWildCam-WILDS, YCB-Objects, and selected negative cases, including Camelyon17-WILDS and some CIFAR-10-C shifts.The trends hold across learning approaches, training durations, and hyperparameters, including application-derived distribution shifts “in the wild.”Source: §8 · p. 16

Related transfer and fine-tuning work

• Related work links predictable generalization to transfer learning and fine-tuning: Kornblith et al. found ImageNet accuracy predicts fine-tuned task accuracy, while Andreassen et al. found fine-tuning increases accuracy but reduces effective robustness.Andreassen et al. study CIFAR-10 and ImageNet, whereas this paper additionally examines three WILDS shifts.Source: §8 · p. 16

9 Discussion

• Linear ID/OOD trends occur across many model types and distribution shifts, but some datasets and adversarial shifts violate them, leaving their scope unresolved.The paper proposes formalizing when OOD performance is approximately determined by ID performance, while noting that current experiments provide evidence rather than proof.Source: §9 · pp. 16–17, §9.2 · p. 19
• The proposed 1%-approximate correlation conjecture applies to shifts including CIFAR-10→CIFAR-10.1, ImageNet→ImageNet-V2, and FMoW-WILDS for models trained by ERM on the in-distribution data.Zero-shot ImageNet models applied to CIFAR-10 follow a different trend, so restricting the conjecture to models trained on in-distribution data is important.Source: §9 · p. 17
• Open questions concern which distribution pairs and model families satisfy the correlation property, including whether it extends beyond ERM to useful models and robustness interventions.Nearest-neighbor models follow ERM trends on several shifts, while interpolation with a random classifier produces models above the trend; the stronger conjecture remains imprecise.Source: §9 · pp. 17–18

9.1 Possible implications

• When correlation holds across related shifts, practitioners can select models using in-distribution performance, while robustness claims should show gains beyond the ERM linear trend.The discussion recommends reporting both ID and OOD performance and illustrating training-technique effects with scatter plots; architecture changes may not reduce the gap, whereas better pre-training data sometimes can.Source: §9.1 · pp. 18–19

9.2 Frequently asked questions

• Not all distribution shifts have linear trends, so improving only in-distribution performance is insufficient; robustness interventions and pre-training remain relevant avenues for reducing OOD gaps.The discussion specifically cites adversarial shifts as counterexamples and identifies pre-training as a promising direction for improving OOD performance.Source: §9.2 · p. 19
• Models can violate linearity after nonlinear accuracy transformations: randomly interpolating a nontrivial model with a random classifier is linear in raw accuracy but not after a probit transform.As the interpolation probability varies, accuracies trace a line from (1/C, 1/C) to the base model’s ID/OOD accuracies, which becomes nonlinear under transformations such as probit.Source: §9.2 · pp. 19–20

A Experimental testbeds

• Three testbeds cover CIFAR-10 and WILDS, ImageNet, and YCB-Objects pose estimation, enabling evaluation across varied datasets and distribution shifts.The CIFAR-10/WILDS testbed is custom-built, the ImageNet testbed is based on Taori et al., and the pose-estimation testbed uses YCB-Objects [17].Source: A Experimental testbeds · p. 32

A.1 CIFAR-10 and WILDS testbed

• The main CIFAR-10/WILDS testbed contains four dataset universes—CIFAR-10, FMoW-WILDS, Camelyon17-WILDS, and iWildCam-WILDS—with datasets for both in- and out-of-distribution evaluation.Each universe contains at least three datasets used to train and test models under different distributions.Source: §A.1 · p. 32

A.1.1 CIFAR-10 and related datasets

• The CIFAR-10 universe tests shifts from reproduction and annotation changes, altered image sources, and synthetic perturbations, including a standard ResNet accuracy drop of 8 to 9 percentage points on CIFAR-10.1.CINIC-10 and STL-10 change the image source to ImageNet, while CIFAR-10-C applies 19 perturbations at five severity levels.Source: §A.1.1 · pp. 32–33

A.1.2 FMoW-WILDS

• FMoW-WILDS classifies 62 land- or building-use categories in satellite images, evaluating average or worst-region accuracy under temporal and geographical shifts.The data span five regions and years from 2002–2013 for training and in-distribution testing, 2013–2016 for out-of-distribution validation, and 2016–2018 for out-of-distribution testing.Source: §A.1.2 · p. 33

A.1.3 Camelyon17-WILDS

• Camelyon17-WILDS evaluates tumor detection in tissue patches, with out-of-distribution performance measuring generalization from three training hospitals to a different hospital.The in-distribution test patches come from the same 30 WSIs across three hospitals, while the out-of-distribution test uses 10 WSIs from another hospital.Source: §A.1.3 · p. 34

A.1.4 iWildCam-WILDS

• iWildCam-WILDS classifies 182 animal species from camera-trap images, using macro F1 to assess generalization across camera traps with differing imaging conditions.Evaluation filtering based on MegaDetector detections improved the observed correlations by one or two R2 points, while rare classes produced fairly large confidence intervals.Source: §A.1.4 · pp. 34–35

A.1.5 Models for CIFAR-10 and WILDS Experiments

• The model suite spans diverse convolutional architectures, ImageNet-pretrained and zero-shot models, and models trained with extra data such as CLIP, robust self-training, and ODST.The testbed includes models trained on CIFAR-10, ImageNet, or additional sources, allowing the experiments to probe architecture, pretraining, and extra-data effects.Source: §A.1.5 · pp. 35–37
• The study adds classical pixel-based methods, including random features, boosting, ridge and logistic regression, support vector machines, discriminant analysis, forests, and nearest neighbors.These methods use standardized image pixels and vary regularization, kernels, tree settings, or neighbor counts where applicable.Source: §A.1.5 · p. 38

A.1.6 Summary statistics

• The testbed trains about 3,000 distinct models and performs about 100,000 evaluations across multiple training and test-set families.Models are evaluated across datasets and, in some cases, intermediate checkpoints to study training-duration effects.Source: §A.1.6 · p. 38, Table 2 · p. 39, Table 3 · p. 39

A.2.1 Datasets

• The ImageNet evaluation suite includes natural shifts such as ImageNet-V2, ObjectNet, video-anchor datasets, and ImageNet-A, while excluding adversarial consistency shifts.The video-anchor datasets use the pm-0 metric defined by Shankar et al..Source: §A.2.1 · pp. 38, 40

A.2.2 ImageNet Testbed Models

• The ImageNet testbed combines existing models with zero-shot CLIP [80], self-supervised models [21], classical low-accuracy models, and low-accuracy ConvNets.The added models broaden variation in supervision, architecture, training duration, and accuracy.Source: §A.2.2 · p. 40

A.3 YCB-Objects Testbed

• The YCB-Objects testbed evaluates 6D pose from 128 × 128 RGB images, measuring correctness when ADD is below 10% of the object-model diameter.ADD averages distances between model points transformed by ground-truth and estimated rotations and translations, following Hinterstoisser et al..Source: §A.3 · p. 40, §A.3.1 · pp. 40–41

A.3.2 YCB-Objects Datasets.

• The synthetic YCB-Objects data uses rendered non-symmetric objects, varied lighting and textures, and disjoint texture subsets to define in-distribution and out-of-distribution tests.Experiments use 50,000-image training sets and 10,000-image in-distribution and out-of-distribution test sets; 100,000-image training is reserved for Appendix B.2.Source: §A.3.2 · p. 41

A.3.3 YCB-Objects Models

• YCB-Objects models predict nine 2D keypoints with semantic-segmentation networks and recover 3D pose using PnP [61].The model family includes UNet encoders and FCN_ResNet backbones with depths 18, 34, 50, and 101.Source: §A.3.3 · pp. 41–42

B The linear trend phenomenon

• Appendix B.1 presents additional linear-trend examples across distribution shifts in CIFAR-10, FMoW-WILDS, ImageNet, and YCB-Objects, while B.2 tests invariance to hyperparameters, training duration, and training-set size.These analyses extend the testbeds introduced in Appendix A.Source: B The linear trend phenomenon · p. 42

B.1.1 CIFAR-10

• R^2 exceeded 0.94 for each of eight selected CIFAR-10-C perturbations, although several other perturbations showed no linear trend.The selected shifts were fog, brightness, snow, defocus blur, spatter, elastic transform, frost, and saturate.Source: §B.1.1 · p. 43
• R^2 exceeded 0.99 for every CIFAR-10.1, CIFAR-10.2, CINIC-10, and STL-10 shift, with linear fits spanning classical and neural models.For STL-10, models used nine-class variants excluding monkey and frog classes.Source: §B.1.1 · p. 42, Figure 7 · p. 43

B.1.2 FMoW-WILDS

• R2 exceeds 0.98 across FMoW-WILDS OOD validation and test sets, using both average and worst-region accuracy after excluding models below 10% ID accuracy.Figure 9 reports linear fits across classical methods and ImageNet networks, with 95% Clopper-Pearson confidence intervals.Source: §B.1.2 · p. 43, Figure 9 · p. 45

B.1.3 ImageNet

• On ImageNet shifts, CLIP models are unusually robust, while low-accuracy models generally remain near the predicted fit except on ObjectNet, YTBB-Anchors, and ImageNet-A.The ImageNet-A fit is performed piecewise around ResNet-50 accuracy.Source: §B.1.3 · p. 45, Figure 10 · p. 46

B.1.4 iWildCam-WILDS

• For iWildCam-WILDS, all final-epoch model error bars intersect the fitted probit-domain trend except AlexNet trained from scratch, while weight decay can affect fine-tuned trends.The experiments vary architecture and learning rate with weight decay fixed to zero in the main comparison.Source: §B.1.4 · p. 47, Figure 11 · p. 48

B.1.5 YCB-Objects

• The synthetic YCB-Objects pose-estimation task shows a strong ID/OOD relationship, with a linear fit achieving R2 = 0.99 under a background-texture shift.The shift changes rendered background textures while holding objects, lighting, and sampling distributions fixed.Source: §B.1.5 · pp. 48, 50, Figure 12 · p. 49

B.2 Variations in model hyperparameters, training duration, and training dataset size

• Across the broader variation studies, changing hyperparameters, training duration, or training-set size generally moves models along established ID/OOD trends without changing their fits.The exploration focuses on CIFAR-10 and FMoW-WILDS to test sensitivity beyond benchmark or synthetic shifts.Source: §B.2 · p. 51, Figure 14 · p. 52, Figure 17 · p. 55

B.2.1 CIFAR-10

• For CIFAR-10.2, CINIC-10, and CIFAR-10-C fog, variation in hyperparameters, training duration, and training-set size preserves the trend, with R2 > 0.99 in every setting.The study spans ridge regression, random features, and ResNet families across low, moderate, and high accuracy regimes.Source: §B.2.1 · p. 51, Figure 14 · p. 52, Figure 15 · p. 52

B.2.2 FMoW-WILDS

• For FMoW-WILDS, these variations preserve the fit with R2 > 0.99 for average accuracy and R2 > 0.91 for worst-region accuracy.The experiments use random forests, random features, and ResNets while varying model settings, training duration, and training-set size.Source: §B.2.2 · p. 54, Figure 17 · p. 55

B.2.3 YCB-Objects

• Increasing YCB-Objects training-set size moves models along the existing trend without changing it, which remains well explained with R2 = 0.98.This result establishes invariance to training-data quantity for the pose-estimation experiment.Source: §B.2.3 · p. 55, Figure 18 · p. 56

C.1 Camelyon17-WILDS

• ID–OOD variability on Camelyon17-WILDS was not explained by training randomness: averaging ten runs gave R^2 = 0.39 versus R^2 = 0.40 without averaging.Early-stopping on a separate OOD validation set produced R^2 = 0.46, while ID-validation early stopping increased correlation to R^2 = 0.77.Source: §C.1 · p. 58, Figure 20 · p. 58
• ImageNet pretraining did not resolve Camelyon17-WILDS variability: convergence-trained models had R^2 = 0.05, rising only to R^2 = 0.14 after seed averaging.The results indicate that averaging pretrained-model runs alone provides little improvement in the observed ID–OOD relationship.Source: §C.1 · p. 58, Figure 21 · p. 59
• R^2 reached 0.95 after targeted color-jitter augmentation and averaging over ten seeds on Camelyon17-WILDS, although all models had relatively high ID accuracy.Color jitter alone achieved R^2 = 0.77 and increased average OOD performance, while seed averaging further reduced OOD fluctuations; whether this holds in the low-accuracy regime remains unclear.Source: §C.1 · p. 59, Figure 21 · p. 59
• Correlated test examples plausibly drive Camelyon17-WILDS variation because patches share slide, hospital, staining, and imaging characteristics, with only 50 slides overall.The OOD test set contains ten slides from one hospital, making evaluation dependence a possible source of instability.Source: §C.1 · p. 59
• In simulated binary CIFAR-10 shifts, increasing effective test size from k = 1–2 to k = 100–500 improved the linear fit from R^2 = 0.66 to R^2 = 0.94 and reduced evaluation variance.By contrast, effective training size had less impact: with k = 2 examples per class, the fit remained fairly precise at R^2 = 0.89.Source: §C.1 · p. 60, Figure 22 · p. 61, Figure 23 · p. 62

C.2 CIFAR-10-C

• For corruption shifts, the strength of the ID–OOD linear trend depended strongly on the corruption, and many corruptions were more linear on ImageNet-C than CIFAR-10-C.The comparison spans neural networks trained on CIFAR-10 or ImageNet and evaluated under related image corruptions.Source: §C.2 · p. 63, Figure 24 · p. 64, Figure 25 · p. 65

C.3 iWildCam-WILDS-v1.0

• The iWildCam-WILDS-v1.0 split produced a far weaker ID–OOD correlation than v2.0, with fine-tuned models near a vertical trend at high ID accuracy and widely varying OOD accuracy.Version 1.0 randomly assigns images from training cameras to train and ID test sets, allowing similar images from the same burst or sequence across those splits.Source: §C.3 · pp. 68–69, Figure 29 · p. 69
• The v1.0 comparison is primarily attributed to the changed train/ID-test split, although the experiments also differed in resolution, learning rates, epochs, and label-noise filtering.The v2.0 split removes image-sequence overlap between train and ID test sets, but near-duplicates remain possible and may contribute to its higher performance variation.Source: §C.3 · p. 69

D.1 Detailed findings for CIFAR-10

• On CIFAR-10, ImageNet-only training lies above the CIFAR-10-trained linear fit, while 80MTI self-training remains on it and ODST deviations change direction across CIFAR-10.1 and CIFAR-10.2.ImageNet-trained models outperform ImageNet zero-shot models but remain closer to the CIFAR-10-trained fit; ODST deviations may reflect auxiliary data or imperfect high-accuracy linearity.Source: §D.1 · p. 70, Figure 30 · p. 71
• CLIP zero-shot and linear-probe models lie above the CIFAR-10 linear trend, especially the higher-performing Vision Transformer.The comparison includes publicly released ResNet-50 and Vision Transformer CLIP models.Source: §D.1 · p. 70
• The CIFAR-10 evaluations implement ImageNet zero-shot classification through a manually constructed class mapping and CLIP zero-shot classification through ensembled text-prompt embeddings.Linear probes fit only the final layer using exact least-squares regression on penultimate-layer activations.Source: §D.1 · p. 71

D.2 Detailed findings for FMoW-WILDS

• On FMoW-WILDS, fine-tuning improves performance without deviating from the ID–OOD line across worst-region and average-region metric combinations.Unlike CIFAR-10, CLIP linear probes do not significantly deviate from the trend, possibly because CLIP’s web data contain fewer satellite-image-relevant examples.Source: §D.2 · pp. 71–72, Figure 31 · p. 72

D.3 Detailed findings for iWildCam-WILDS

• For fine-tuned iWildCam-WILDS models, nonzero weight decay shifts points above the zero-weight-decay ID–OOD trend.For models trained from scratch, macro-F1 measurement error prevents a confident conclusion about weight decay’s effect.Source: §D.3 · p. 72, Figure 32 · p. 73
• Despite this shift, the best iWildCam-WILDS models in both ID and OOD performance use no weight decay.Increasing weight decay appears to move models above the zero-weight-decay line, but not to the highest-performing region.Source: §D.3 · pp. 72–73

E.1 Proof of Theorem 1

• Theorem 1 establishes a high-probability linear trend between ID and OOD probit accuracies when the shift Δ is independent of the classifier.A sphere concentration bound controls the classifier–shift inner product, and a union bound extends the guarantee to multiple classifiers.Source: §E.1 · pp. 73–74
• The theorem applies to any classifier trained solely on samples from D, including the linear classifiers used in the simulations.Such classifiers are independent of Δ by construction.Source: §E.1 · p. 74

E.2 Departures from the linear trend

• Adding samples from the broader distribution D2 improves OOD performance but worsens ID performance, producing separate empirical trends while escaping the theorem’s guarantee.The training samples from D2 depend on Δ, so classifier–shift independence no longer holds.Source: §E.2 · p. 75
• An adversarial shift aligned with a target classifier can substantially reduce its OOD performance and break the linear trend.Choosing Δ = −0.03θ*/||θ*|| produces a particularly strong breakdown for the targeted classifier; this shift matches the Gaussian adversarial-example model of Schmidt et al. [93].Source: §E.2 · p. 75
• Non-isotropic noise covariance breaks the linear relationship because ID accuracy favors small covariance-weighted norms, whereas OOD accuracy also depends on the Euclidean norm.The simulations show stronger ℓ2 regularization improves OOD performance, while ℓ1-regularized logistic classifiers do not achieve good OOD performance.Source: §E.2 · p. 76

E.3 Simulation details

• The simulations vary logistic, ridge, nearest-neighbor, and random-forest models across regularization, training-set size, dimensionality, and selected estimator counts.Most experiments use training subsets of 30–100 samples and projected dimensions from 50 to 3000; the covariance experiment uses 100–2000 samples and dimension 500.Source: §E.3 · p. 76
• Linear-model accuracies are computed exactly, whereas other models use sampled estimates with 95% Clopper–Pearson confidence intervals.Therefore, error bars are omitted for linear models and shown for the remaining models.Source: §E.3 · p. 76
• The standard simulations use a random unit-vector mean in dimension 10^5 with σ = 10^-1.5, α = 0.7, β = 0.5, and γ = 1, while covariance simulations use dimension 500 and a diagonal heterogeneous covariance.The covariance matrix has 490 entries of 1/2 and the remainder of 1/200, with small entries placed randomly.Source: §E.3 · pp. 76–77

PAC-Bayesian analysis of distribution shift

• PAC-Bayesian bounds relate OOD performance to target divergence, disagreement, or joint source error, but do not explain the paper’s observed bidirectional linear ID–OOD trends.Li and Bilmes do not directly relate source and target error, while Germain et al. [39] use quantities differing from ID performance or provide only one-sided bounds.Source: F Additional related work · p. 77

Reliability of machine learning benchmarks

• Precise linear trends can make CIFAR-10 and ImageNet benchmarks more reliable for comparing models, even when absolute accuracy is unstable across OOD test sets.The methodology treats OOD evaluation as testing stability under perturbations to the test set: accuracies consistently drop, but model rankings remain stable.Source: F Additional related work · p. 77

Theoretical models for linear trends in earlier work on dataset reproduction

• Earlier models explain linear fits through easy–hard mixtures or continuous image hardness, but neither they nor the paper’s model predicts which real distribution shifts will produce linear trends.Recht et al. [85] require linear axis scaling, whereas Recht et al. [86] obtain closer fits in the probit domain; the paper’s concrete generative model also supports departures from linearity.Source: F Additional related work · pp. 77–78

Linear trends in image classification with natural language supervision

• Under CLIP training, models perform substantially above the ImageNet distribution-shift linear trend, while varying architecture and training compute produces a clear linear trend across ResNet variants and Vision Transformers [37].Source: F Additional related work · p. 78

Underspecification as defined in D’Amour et al. [30]

• Evidence on underspecification is nuanced: some ImageNet-C and CIFAR-10-C shifts vary substantially, ObjectNet remains predictable overall, and medical-imaging results differ across datasets.D’Amour et al. [30] report ophthalmological evidence of underspecification, whereas dermatological evidence is less clear, with p-values 0.54, 0.42, 0.29, and 0.03; overall, some shifts correlate more strongly than others.Source: F Additional related work · p. 79

Further distribution shifts without universal linear trends

• Adversarial shifts are a clear counterexample to universal linearity, while distribution-shift surveys and robustness methods motivate improving over empirical risk minimization when target-domain adaptation data are unavailable.Standard CIFAR-10 models can have 0–10% robust accuracy versus about 65% for robustness-trained models at 91% standard accuracy; domain-generalization surveys find methods often match or underperform ERM.Source: F Additional related work · pp. 79–80