Source-linked AI summary
Why do classifier accuracies show linear trends under distribution shift?
Horia Mania, Suvrit Sra
TL;DR
Models’ accuracies on original and shifted data have been observed to follow approximately linear trends, raising questions about why and when this happens. The paper analyzes these trends under a model-similarity assumption and finds approximate collinearity unless the distribution shift is large. It connects understanding these trends to reasoning about robustness.
Problem
Studies of newly collected test data report approximately linear accuracy trends across distributions, raising the question of why they occur and when to expect them.
Source: §1 · p. 1Method
Results
Models must be approximately collinear across two distributions under the assumption unless the distribution shift is large in a certain sense.
Source: §1 · p. 2Takeaways & Limitations
Understanding when models are collinear is presented as useful for reasoning about robustness, for which the paper describes non-collinearity and proximity to the identity line as relevant.
Source: §1 · p. 1Assumption 1 alone does not guarantee collinearity; suitable closeness between distributions P and Q is also needed.
Source: §2 · p. 3
Abstract
from arXiv · showhide
Recent studies of generalization in deep learning have observed a puzzling trend: accuracies of models on one data distribution are approximately linear functions of the accuracies on another distribution. We explain this trend under an intuitive assumption on model similarity, which was verified empirically in prior work. More precisely, we assume the probability that two models agree in their predictions is higher than what we can infer from their accuracy levels alone. Then, we show that a linear trend must occur when evaluating models on two distributions unless the size of the distribution shift is large. This work emphasizes the value of understanding model similarity, which can have an impact on the generalization and robustness of classification models.
1 Introduction
Recent studies test whether models have overfit popular datasets by recreating data-collection pipelines and evaluating models on newly collected data. These evaluations reveal approximately linear accuracy trends across distributions, motivating an analysis of their cause and conditions.
1 Introduction
- New-test accuracies are approximately linear functions of original-test accuracies, despite observed drops of 3%–15% on CIFAR-10 and 11%–14% on ImageNet.The pattern has also been observed on question-answering data [15] and other ImageNet variants [23].Source: §1 · p. 1
- The paper asks why this collinearity occurs across two distributions and when it should be expected.Source: §1 · p. 1
- Understanding collinearity is motivated by robustness: the paper describes a robust model as one that is not collinear with prior models and lies near the identity line.Source: §1 · pp. 1–2
- The proposed explanation assumes that lower-accuracy models’ correct predictions are usually also correct for higher-accuracy models, an assumption reported for ImageNet and CIFAR-10 models.This premise is inspired by Mania et al. [14].Source: §1 · p. 2
- Under the similarity assumption, models must be approximately collinear across two distributions unless the distribution shift is large in a certain sense.Understanding this pattern is presented as useful for reasoning about model robustness.Source: §1 · pp. 1–2
2 Main results
The analysis combines a model-ordering condition based on prediction similarity with a restricted notion of distributional closeness. Together, these conditions guarantee approximate collinearity unless the shift is large, while examples show neither condition suffices alone.
2 Main results
- The analysis relies on models having more similar predictions than their accuracy levels alone imply; dominance probability measures when a lower-accuracy model is correct and a higher-accuracy model is wrong.This premise draws on prior empirical observations for ImageNet and CIFAR-10 models.Source: §2 · p. 3
2.1 Characterizing distributional closeness
- The paper’s closeness condition constrains selected model-dependent events, so distributions may still be far apart under traditional distances such as total variation or KL.The condition requires small probability differences only for a restricted set of events.Source: §2.1 · pp. 3–4
- Applying the bound to multiple model triples can narrow the feasible region and support a better piecewise-linear fit when one line leaves large residuals.The analysis suggests two-piece regression as an option in such cases.Source: §2.1 · p. 4
- For an ordered model set, distributional closeness guarantees a line with bounded residuals; smaller shifts and narrower accuracy ranges tighten the bound.The guarantee applies to all models, and the residual bound depends on the maximum accuracy difference within the set.Source: §2.1 · pp. 4–5
2.2 Examples and Notation
- Even when Assumption 1 orders models under both P and Q, it alone does not guarantee that classification accuracies lie close to a line; suitable closeness between P and Q is also needed.In the example, accuracies under Q are 0.5, 0.9, and 0.9.Source: §2 · p. 3, §2.2 · p. 6
- 0.163 residual versus a 0.055 bound: a (0.31, 0.38, 0.005, 0.008)-close shift can still break collinearity when models are unordered.The example shows that distributional closeness alone is insufficient at these parameter values.Source: §2.1 · p. 4, §2.2 · p. 7
3 Model dominance
The section uses a dominance probability to express model similarity in a form suited to the analysis, then checks that quantity across ImageNet models. The measured probabilities are generally small and decline as accuracy differences grow.
3 Model dominance
- Mania et al. [14] define model similarity as the probability that two models agree on whether a data point is classified correctly.Under independent errors, this probability is µiµj+(1−µi)(1−µj).Source: §3 · p. 7
- Dominance probability measures the chance that a lower-accuracy model is correct while a higher-accuracy model is incorrect.The authors use this quantity instead of model similarity, which they find difficult to work with because its level depends on model accuracies.Source: §3 · p. 7
- For 66 ImageNet models, 76% of dominance probabilities are below 0.05 and all are below 0.09.The probabilities decrease as model accuracy differences increase; the authors infer that more accurate models tend to dominate lower-accuracy models.Source: §3 · p. 8
4 The size of distribution shift
The section measures distributional closeness using bounds on a finite set of events, which can be estimated empirically. For ImageNet and ImageNetV2, the measured events fit within a wedge, and most fit within a smaller one.
4 The size of distribution shift
- The closeness condition need only hold for a finite set of events, and relevant event probabilities can be estimated empirically.Fewer required events make the closeness condition more permissive.Source: §4 · p. 8
- The two correctness-outcome events omitted from the six-event condition shift all three model points equally and therefore do not affect their closeness to a line.With three models there are eight possible correctness outcomes, but the definition considers six.Source: §4 · p. 8
- Some event-bound violations can be tolerated when other triplets connect the models, and the analysis can also use nonlinear bounds on distribution differences.For four models, bounds for triplets (f1, f2, f4) and (f1, f3, f4) can place f2 and f3 near the line through f1 and f4.Source: §4 · p. 8
- For this dataset, linear bounds require large δ2 and ν2 even though the relevant event probabilities differ little between distributions.The authors discuss this limitation further in Appendix A.Source: §4 · p. 9
- For 274,560 model-triplet events, ImageNet and ImageNetV2 are empirically (0.31, 0.38, 0.005, 0.008)-close.The events lie within a wedge with slopes 0.69 and 1.38 and y-intercepts −0.005 and 0.008.Source: §4 · p. 9
- 95% of the 274,560 events fit within a smaller wedge using δ1 = 0, δ2 = 0.25, ν1 = 0.005, and ν2 = 0.005.The authors connect this observation to their later analysis of linear fit in the probit domain.Source: §4 · p. 9
5 Main arguments
The section derives residual bounds for ordered and approximately ordered models, then compares those bounds with probit and piecewise-linear fits on ImageNet. It argues that the observed probit fit is compatible with broader trends allowed by the analysis.
5.1 Proof of Proposition 1
- A line through the midpoints of the two endpoint segments halves the residual for the middle model compared with the line through the endpoints.The residual is measured vertically from the middle model to the line joining the lowest- and highest-accuracy models.Source: §5.1 · pp. 9–10
5.2 Approximately ordered models
- For models that are not ordered, the analysis allows small nonzero dominance probabilities and numerically bounds the residual under Assumption 1.The numerical method grids over probabilities subject to the assumption and distribution-closeness constraints.Source: §5.2 · p. 10
- 0.0225: with δ1 = 0, δ2 = 0.25, and ν1 = ν2 = 0.005, a line can bound the three models’ residuals at most 0.0225.The numerical bounds do not degrade significantly when models satisfy Assumption 1 with small ζ rather than being ordered.Source: §5.2 · pp. 10–11
5.3 Discussion of probit axis scaling
- With tighter closeness parameters, the lower bounds closely follow the probit curve, suggesting the fit may result from models saturating the bounds.The plotted lower bounds use δ1 = 0, δ2 = 0.25, and ν1 = ν2 = 0.005; the earlier parameter choice is described as conservative.Source: §5.3 · pp. 11–12
- 0.997: a two-piece ImageNet fit nearly matches the probit fit’s R2 of 0.998, with a difference of 4 × 10^-4.The paper notes that piecewise-linear fits may occur more generally than probit-linear fits.Source: §5.3 · pp. 11–12, Figure 5 · p. 12
6 Conclusion
The conclusion distinguishes ways a new model may exhibit robustness relative to existing accuracy trends, including by revealing that distributions are not close or by differing from prior models. It leaves open when the sufficient conditions apply and why models are similar.
6 Conclusion
- A new model that improves accuracy but remains near the previous trend has relative robustness, not effective robustness.This distinction follows Taori et al.’s terminology for models evaluated against the existing linear trend.Source: §6 · p. 12
- A new model can be effectively robust by revealing that the distributions are not close, potentially when it is trained on a different dataset.In that case, relevant events on which the new model is correct can have much higher probability under Q than under P.Source: §6 · pp. 12–13
- A model dissimilar to previous models can lie above or below their line even when the distributions are close; the authors conjecture CLIP may be such a case.They leave analysis of CLIP for future work.Source: §6 · p. 13
- The paper gives sufficient conditions for approximate collinearity, but whether they apply in practice and why models are similar remain open questions.The authors specifically ask whether similarity comes from training data, model classes, or something else.Source: §6 · p. 13
A CIFAR-10.1 vs CIFAR-10
The CIFAR-10 and CIFAR-10.1 models empirically satisfy the paper’s distribution-closeness assumption, although the bound is loose for events with low CIFAR-10 probability. Dominance probabilities are also small across CIFAR-10 model pairs.
A CIFAR-10.1 vs CIFAR-10
- CIFAR-10 and CIFAR-10.1 satisfy the assumption with δ1 = 0.0, δ2 = 1.7, ν1 = 0.005, and ν2 = 0.02, based on 30 models.The evaluation considers probabilities of events defined by model triplets.Source: A CIFAR-10.1 vs CIFAR-10 · p. 16
- For events with CIFAR-10 probability at least 0.03, δ2 = 0.8 and ν2 = 0.01 satisfy the bounds for over 95% of events.The larger δ2 = 1.7 is needed only for events with CIFAR-10 probabilities at most 0.03; using it in the guarantee gives a poor bound.Source: A CIFAR-10.1 vs CIFAR-10 · p. 16
- The analysis can tolerate some events violating the bounds, and distribution closeness can use non-linear bounds on Q(A) − P(A).The bounds may take the form −g1(P(A)) ≤ Q(A) − P(A) ≤ g2(P(A)).Source: A CIFAR-10.1 vs CIFAR-10 · p. 16
- All CIFAR-10 model-pair dominance probabilities are below 0.08, and most are at most 0.04, supporting Assumption 1 with small ζ.Figure 6b displays the dominance probabilities.Source: A CIFAR-10.1 vs CIFAR-10 · p. 16
B Proof of Corollary 1
The proof bounds the maximum residual of the models from a common line by considering the points farthest above and below a line through the endpoint models. It concludes with a constant that the authors note may be improved.
B Proof of Corollary 1
- If all models lie on or below the endpoint line, a line through the midpoints of the two segments around the point with largest residual limits every residual to at most half that point’s residual.The harmonic mean of two numbers with constant sum is maximized when they are equal.Source: B Proof of Corollary 1 · p. 17
- If points lie on both sides of the endpoint line, the largest residuals above and below it bound all other points between parallel lines, allowing a center line with residual at most (rD + rB)/2.Here rB and rD are the residuals of the extremal points below and above the endpoint line.Source: B Proof of Corollary 1 · p. 17
- The proof optimizes the residual bound over nonnegative segment lengths a, b, and c whose sum is the endpoint accuracy difference.The optimization uses first-order conditions after combining the preceding bounds.Source: B Proof of Corollary 1 · p. 17
- The proof’s constant 25/64 could be improved with a more involved argument.The result follows by choosing a line parallel to AC with equal residuals to B and D.Source: B Proof of Corollary 1 · p. 17
C The missing Fisher vector model
Including the omitted 64-center Fisher vector model causes some events to fall outside the original closeness bounds, but does not violate Assumption 1. The authors omit it for clarity because it makes ImageNet and ImageNetV2 appear farther apart under their measure.
C The missing Fisher vector model
- The analysis uses 66 of the 67 models collected by Recht et al., omitting the Fisher vector model with 64 Gaussian Mixture Model centers.The appendix explains this choice by comparing the analysis with that model included.Source: C The missing Fisher vector model · pp. 17–18
- The authors omit the 64-center model because it would make ImageNet and ImageNetV2 appear far apart under Definition 1, although its inclusion does not violate Assumption 1.They note that further investigation is needed to explain why this model defines events whose probabilities shift so much.Source: C The missing Fisher vector model · p. 18
- With the 64-center Fisher vector model included, the original wedge does not cover all 287,430 events defined by the 67 models.The wedge uses δ1 = 0.31, δ2 = 0.38, ν1 = 0.005, and ν2 = 0.008.Source: C The missing Fisher vector model · p. 18, Figure 7 · p. 18
- 666 of the 12,870 events introduced by the omitted Fisher vector model fall outside the wedge, while only 195 lie in three clusters far from the identity line.The remaining new points fall close to the origin.Source: C The missing Fisher vector model · p. 18