Understanding the Failure Modes of Out-of-Distribution Generalization

Vaishnavh Nagarajan; Anders Andreassen; Behnam Neyshabur

Source-linked AI summary

Understanding the Failure Modes of Out-of-Distribution Generalization

Vaishnavh Nagarajan, Anders Andreassen, Behnam Neyshabur

arXiv:2010.15775v3Oct 29, 2020cs.LGcs.CVstat.ML

TL;DR

ERM can rely on spurious features even when invariant features fully predict the label, creating out-of-distribution failures that simpler explanations do not cover. The paper studies gradient-descent-trained linear classifiers on constrained easy-to-learn tasks, identifies geometric and statistical skews, and tests them across image datasets and neural networks. Within the studied linear setting, the authors report that these skews account for the considered failure modes, while more complex nonlinear real-world tasks may exhibit others.

Problem
The paper asks why ERM relies on spurious features and fails under distribution shifts even when invariant features fully predict labels.
Method
The authors theoretically study gradient-descent-trained linear classifiers on constrained easy-to-learn tasks and empirically test the resulting mechanisms on image datasets and neural networks.
Results
The paper identifies two complementary ERM failure mechanisms caused by geometric and statistical skews, and validates these skews across MNIST- and CIFAR10-based tasks and neural networks.
Takeaways & Limitations
Within easy-to-learn tasks and gradient-descent-trained linear models, the absence of geometric and statistical skews coincides with no reliance on spurious features.
Takeaways & Limitations
More complicated tasks and nonlinear deep models can exhibit additional failure modes beyond the mechanisms identified here.

Abstract

from arXiv · show

Empirical studies suggest that machine learning models often rely on features, such as the background, that may be spuriously correlated with the label only during training time, resulting in poor accuracy during test-time. In this work, we identify the fundamental factors that give rise to this behavior, by explaining why models fail this way {\em even} in easy-to-learn tasks where one would expect these models to succeed. In particular, through a theoretical study of gradient-descent-trained linear classifiers on some easy-to-learn tasks, we uncover two complementary failure modes. These modes arise from how spurious correlations induce two kinds of skews in the data: one geometric in nature, and another, statistical in nature. Finally, we construct natural modifications of image classification datasets to understand when these failure modes can arise in practice. We also design experiments to isolate the two failure modes when training modern neural networks on these datasets.

1 INTRODUCTION

The paper studies why ERM relies on spurious features and fails under distribution shifts even when invariant features fully predict labels. It uses easy-to-learn tasks to identify two complementary skew-based failure mechanisms and tests them theoretically and empirically.

Motivation: ERM can exploit training-only correlations such as background features, causing accuracy to deteriorate when those correlations shift at test time.Domain generalization seeks robustness to shifts across unseen domains.
Motivation: Existing explanations do not account for ERM relying on spurious features when invariant features are fully predictive, including in real-world-like tasks.The paper highlights this as a formal gap in understanding easy-to-learn out-of-distribution failures.
Motivation: The analysis aims to provide a foundation for explaining when and why different out-of-distribution generalization algorithms may succeed or fail.This motivation follows from the limited theory available for comparing ERM with alternatives.
Contributions: The authors formulate easy-to-learn tasks in which invariant features are fully predictive, creating theoretical and simplified empirical test beds for studying ERM failure.These tasks include simplified MNIST- and CIFAR10-based classification settings.
Contributions: Theoretical analysis of gradient-descent-trained linear classifiers identifies two complementary ERM failure mechanisms caused by geometric and statistical skews.The analysis includes logistic or exponential loss and the infinite-time max-margin limit.
Contributions: Experiments validate the skews across MNIST- and CIFAR10-based tasks and demonstrate their effects in fully connected networks and ResNets.The study also examines failures beyond standard spurious-correlation settings and includes a non-image classification experiment.

2 RELATED WORK

Prior work explains spurious-feature reliance either through partially predictive features or through simpler-to-learn spurious features. This paper argues that neither account explains failures in its easier task class and develops a broader explanation.

Existing explanations: One existing model assumes invariant and spurious features are only partially predictive, preventing an accuracy-maximizing classifier from ignoring the spurious feature.This account makes the task difficult enough that spurious reliance can improve training accuracy.
Paper's distinction: The paper studies easier tasks where invariant features are fully predictive and linearly separable, while spurious features are not fully predictive.Its analysis also does not require the spurious feature to depend on the invariant feature.
Paper's distinction: The analysis extends beyond max-margin settings to finite-time gradient-descent classifiers trained with logistic loss.This addresses training dynamics rather than only the infinite-time limit.
Paper's distinction: The paper positions its explanation as more fundamental while noting concurrent work that instead emphasizes overparameterization and requires dependence between spurious and invariant features.The authors state that their results are agnostic to overparameterization and do not require that dependence.
Algorithms: Sophisticated domain-generalization methods commonly constrain representations or feature distributions across domains, but prior comparisons do not establish a substantial improvement over ERM.The paper cautions that this observation does not excuse ERM's failures.

3 EASY-TO-LEARN DOMAIN GENERALIZATION TASKS

The paper defines constrained domain-generalization tasks that remain easy because invariant features are informative and stable, then shows how ERM can nevertheless fail on a Binary-MNIST construction. These constraints exclude several alternative sources of failure.

Setting: Domain generalization pools training data from multiple domains into D_train, while invariant and spurious features are defined relative to a robust classifier over possible domains.ERM minimizes average loss on the pooled training distribution.
Empirical setup: In the Binary-MNIST task, random ReLU features make invariant representations linearly separable, while a training-time background correlation is flipped completely at test time.The construction uses the first five versus remaining five digits as classes and sets Pr_Dtest[xsp · y > 0] = 0.0.
Task design: The easy-task framework is designed to prevent complex examples from making ERM rely on spurious features for non-fundamental reasons.The resulting class serves as a theoretical and simplified empirical test bed.
Constraints: The tasks require identical marginal distributions of invariant features across domains, so domain shifts occur through other features.The paper notes that unstable invariant distributions would make ERM easier to fail.
Constraints: They impose conditional independence of spurious and invariant features given the label, excluding failures based on complex relationships between background and object shape.This mirrors independently selecting the color feature after choosing the MNIST class and digit.
Constraints: Spurious features are restricted to two training values, +B and −B, excluding failures driven by high-dimensional or carefully constructed continuous-valued features.The discrete restriction applies during training, while test-time shifts may remain within or leave that support.
Constraints: The identity representation x = (xinv, xsp) excludes failures caused by a hard-to-disentangle representation.The analysis would also hold under orthogonal transformations because the studied algorithms are rotation-invariant.
Empirical setup: The simplest four-point easy-to-learn task initially shows no reliance on the spurious feature despite its correlation, motivating the paper's search for additional failure factors.The four-point construction satisfies the stated invariant, independence, and two-valued-spurious-feature conditions.

4 FAILURE DUE TO GEOMETRIC SKEWS

Geometric skews make invariant-only classification require larger norms than combining a spurious shortcut with a low-norm minority separator, so max-margin classifiers use the spurious feature. The mechanism extends beyond linear theory to several image-task failures, while the increasing-norm premise remains unexplained.

Geometric skew: As training samples increase, the least-norm max-margin classifier using only invariant MNIST features has a growing ℓ2 norm.The paper takes this increasing-norm behavior as an observed geometric property underlying the analysis.
Geometric skew: A geometric skew is the ℓ2-norm gap between the all-data invariant separator and a low-norm separator for the minority group.The dataset is divided into majority and minority groups according to whether the spurious feature agrees with the label.
Failure mechanism: Because the minority separator has much smaller norm, max-margin classification combines the spurious feature for the majority with that separator for the minority.This produces a diagonal decision boundary rather than a purely invariant vertical separator.
Formal result: Theorem 1 bounds the spurious weight more strongly as the minority-to-all invariant norm ratio shrinks, formalizing when geometric skew induces spurious reliance.The informal theorem applies to linear classifiers on easy-to-learn tasks with B = 1.
Formal result: As training correlation approaches 1.0, the spurious weight approaches ≈1 and sign flips can make the margin negative; as correlation approaches 0.5, the spurious component shrinks toward ≈0.The result links minority-group size, norm ratios, spurious reliance, and vulnerability to test-time shifts.
Broader empirical implications: The geometric argument also accounts for failures in nonlinear or non-easy image tasks, including CIFAR10 and colored Cats-vs.-Dogs modifications.The paper reports accuracy drops of ≳10% and ≳27% in specific ResNet experiments, while noting one task breaks a constraint nonessentially.
Broader empirical implications: These examples broaden how spurious correlations can manifest and how varied failure modes can be reasoned about geometrically.The paper presents the examples as empirical implications of the theoretical geometric argument.

5 FAILURE DUE TO STATISTICAL SKEWS

Even without geometric skew, finite-time gradient descent can retain spurious reliance because training updates absorb statistical correlation early and later gradients decay too quickly to remove it. Experiments with neural networks show greater vulnerability after training on statistically skewed data.

Statistical skew: Gradient descent converges to max-margin only asymptotically under the stated separability conditions, so finite-time models can suffer statistical-skew failure.The analysis studies linear classifiers trained with logistic or exponential loss.
Statistical skew: Although max-margin classification ignores the spurious feature in the simplest four-point task, finite-time gradient descent can retain a spurious component.The component stagnates around a value proportional to the training correlation even after long training.
Formal result: Theorem 2 derives a distribution-specific convergence bound for the spurious-to-invariant output ratio on easy-to-learn datasets without geometric skews.The result uses continuous-time gradient descent under exponential loss and allows higher-dimensional invariant features.
Failure mechanism: Early updates absorb spurious correlation in proportion to (2p − 1)B, while exponentially decaying gradients require exponentially long time to remove it.The absorbed component is small near p ≈0.5 and larger near p ≈1.
Broader empirical implications: In neural-network experiments, training on statistically skewed Sexp produces models more vulnerable than no-skew Scon to test-time shifts in spurious-feature scale and correlation.The comparison is shown for fully connected Binary-MNIST and ResNet CIFAR10 tasks.
Broader empirical implications: The experiment isolates statistical skew by comparing controlled datasets designed to separate it from geometric skew, which commonly co-occurs in gradient-descent training.The paper describes disentangling the two effects as challenging in practice.

6 CONCLUSIONS AND FUTURE WORK

The paper identifies geometric and statistical skews as complementary causes of ERM reliance on spurious correlations in easy-to-learn settings, while finding no single failure mechanism even for simple tasks. It argues that more complicated tasks and deep models introduce additional failure modes, motivating further work on unified solutions.

Conclusions: Geometric and statistical skews provide two complementary ways for gradient-descent-trained ERM models to rely on training spurious correlations.When these skews disappear, the paper reports no failure for the studied easy-to-learn linear models.
Conclusions: For easy-to-learn tasks and gradient-descent-trained linear models, the discussion likely captures all possible failure modes when the identified skews are absent.This is presented as a scope-limited conclusion rather than a claim about all models or tasks.
Limitations and scope: More complicated real-world tasks and nonlinear deep models can exhibit additional failure modes beyond the fundamental geometric and statistical mechanisms.The paper explicitly cautions that the broader setting is not governed by only the analyzed mechanisms.
Future work: Even simple tasks can fail through more than one mechanism, so future OoD work should examine whether a unified solution can address all failure mechanisms.The paper presents this as a practical implication and points to outlined solutions as a starting point.

A CONSTRAINTS AND OTHER FAILURE MODES

The paper shows that ERM can fail even when invariant features are fully predictive, and uses violated task constraints to isolate additional failure modes.

Weakly predictive invariant feature: Breaking invariant-feature informativeness lets the Bayes-optimal training classifier rely on spurious features when invariant noise is substantial.As invariant noise vanishes, the Bayes-optimal classifier approaches the invariant feature alone.
Unstable invariant feature: Breaking stability of invariant features causes test failures when examples move closer to the true decision boundary.Training examples at xinv = 2y or 3y induce a boundary at zero, misclassifying positive test examples near xinv = −0.5.
Complex conditional dependencies: Breaking conditional independence allows training-time geometric relationships between invariant and spurious features to make the max-margin classifier rely on the spurious feature.Even a fully informative invariant feature does not prevent failure when xinv + xsp determines the class during training.
High-dimensional spurious features: High-dimensional spurious features can dominate max-margin classification with high probability when many weakly predictive coordinates are available.The construction uses D binary spurious coordinates whose individual label correlations exceed one-half by a margin c.
Hard-to-separate features: Non-orthogonal feature representations create failure because a fully label-informative combined coordinate also contains the shifting spurious component.A CIFAR10 modification adds a line whose brightness changes at test time.

B.1 PROOF OF THEOREM 1 ON FAILURE DUE GEOMETRIC SKEWS

The geometric-skew analysis relates spurious reliance to differences in least-norm invariant classifiers across minority, majority, and full datasets.

Definitions: Geometric skew is the gap between least-norm invariant classifier norms for the minority subset and the full or majority dataset.The minority norm is smaller because the minority subset is smaller, and the ratio controls spurious reliance.
Theorem 3: Theorem 3 parameterizes geometric skew using κ1, κ2, ˜κ1, and ˜κ2 and bounds the spurious component of the max-margin classifier.The constants c1 and c2 depend on B and the least-norm classifiers for the full and majority datasets.
Interpretation: The lower bound is intentionally not numerically tight; it is designed to make the effect of geometric skew instructive.The authors avoid tightening several proof steps to keep the result and proof simple.
Lower bound: A sufficiently large geometric skew forces a sufficiently large positive spurious component in the overall max-margin classifier.The proof constructs a classifier using the spurious feature to obtain a large margin and lower-bounds |Bwsp|.
Upper bound: When geometric skew is negligible, the spurious component must be sufficiently small under the theorem’s task constraints.The upper-bound argument compares the invariant norm required on the negatively affected group with the full-dataset least norm.

B.2 PROOF OF THEOREM 2 ON FAILURE DUE TO STATISTICAL SKEWS

The statistical-skew analysis studies gradient descent when invariant-feature distributions match across majority and minority groups but spurious correlation frequencies differ.

Setup: Theorem 4 assumes identical invariant-feature distributions across majority and minority subsets and analyzes exponential-loss gradient descent from the origin.The setting uses infinitesimal learning rate and a spurious-correlation level p in [0.5, 1).
Invariant dynamics: Gradient descent’s invariant component grows logarithmically while its residual remains smaller-order under the max-margin asymptotics.For sufficiently large t, the invariant margin lies between 0.5 ln(1 + t) and 2M ln(1 + t).
Spurious dynamics: The spurious coefficient is dynamically bounded above by the equilibrium value 1/(2B) ln(p/(1−p)).Starting from zero, the update cannot cross this point because its derivative changes sign there.
Spurious dynamics: The proof obtains a lower bound on the spurious-to-invariant ratio by integrating a lower bound on the spurious update.The argument combines bounds on both components after controlling the invariant margin.

B.3 PRECISE ANALYSIS OF STATISTICAL SKEWS FOR A 2D SETTING UNDER EXPONENTIAL LOSS

The 2D exponential-loss analysis gives explicit time-dependent expressions for invariant and spurious weights as functions of the empirical correlation level p.

Setting: The explicit solution is derived for a 2D dataset with spurious-feature scale B = 1 under infinitesimal-learning-rate exponential loss.A corresponding logistic-loss analysis follows separately.
Explicit dynamics: The dynamics satisfy winv + wsp = ln(1 + 2pt) and winv − wsp = ln(1 + 2(1−p)t).These relations separate majority and minority contributions to the two learned components.
Explicit dynamics: Consequently, winv is the average of the two logarithmic terms, while wsp is half their difference.The spurious coefficient grows with the imbalance between majority and minority correlation frequencies.
Relative reliance: The analysis studies β(t) = wsp/winv to characterize the relative strength of spurious and invariant components over time.The proof separately bounds this ratio for t ≥ 1 and t < 1.

B.4 ANALYSIS OF STATISTICAL SKEWS FOR A 2D SETTING UNDER LOGISTIC LOSS

Under logistic loss in the 2D setting, the classifier dynamics bound the invariant and spurious weights over time and characterize their relative growth when the spurious feature is statistically skewed.

Setup: The logistic-loss analysis studies a linear classifier initialized at the origin on the 2D task, with p denoting the probability that the spurious feature agrees with the label.The stated setting assumes infinitesimal learning rate and sufficiently large training time.
Dynamics: The loss combines contributions from examples aligned and misaligned with the spurious feature, weighted by p and 1 − p.Its two terms depend on the invariant and spurious weights through winv + wsp and winv − wsp.
Bounds: For all training times, the invariant weight is nonnegative and bounded above by ln(t + 1), while the spurious weight cannot become negative when p ≥ 0.5.The nonnegativity follows because the derivative of wsp at wsp = 0 is nonnegative.
Proof strategy: The proof rewrites the dynamics for winv + wsp and winv − wsp, then obtains separate logarithmic expressions for winv and wsp.Adding and subtracting the two relations yields the displayed evolution equations for the invariant and spurious components.
Relative growth: The resulting lower- and upper-bound arguments control wsp(t)/winv(t) by separately lower-bounding wsp and winv.The proof handles the cases wsp > winv and wsp ≤ winv before combining the bounds.

C.1 RANDOM FEATURE EXPERIMENTS FROM SECTION 3.1

Random-feature experiments use highly overparameterized representations to test whether ERM-trained classifiers can exhibit out-of-distribution accuracy drops on Binary-MNIST and two-class CIFAR10.

Random-feature setup: The Binary-MNIST experiment represents each image with 50k random ReLU features, making the model overparameterized for datasets up to 6400 examples.The representation is xinv = ReLU(Wxraw), with Gaussian entries in W.
Spurious feature: The spurious feature takes values −100 or 100, approximately one-tenth the magnitude of the invariant representation norm.The experiment uses this feature support in the Binary-MNIST task.
CIFAR10 validation: A two-class CIFAR10 experiment also reports an out-of-distribution accuracy drop for the random-features model.It uses the first two CIFAR10 classes because grouping five classes per class gives poor in-distribution accuracy initially.

C.2 INCREASING NORM EXPERIMENTS FROM SECTION 4

The experiments validate increasing-norm behavior in MNIST and CIFAR10 and construct CIFAR10 examples showing how non-orthogonal or statistically structured spurious features can produce OoD failures.

Increasing-norm experiments: Increasing the number of datapoints requires larger norms for random-feature models to fit Binary-MNIST and two-class CIFAR10.This observation is the empirical premise of the geometric-skew argument.
Neural-network proxy: Neural-network experiments use distance traveled from initialization as a proxy for the undefined margin, with FNN and ResNet measurements presented for MNIST and CIFAR10.The proxy is used because neural-network margins are not well-defined.
Optimization effects: Fixed-learning-rate SGD recovers increasing norms on CIFAR10, whereas Adam with a decaying learning rate produces norm saturation after a point.Some SGD runs reach 99% rather than 100% accuracy, so the final-epoch norm is reported.
CIFAR10 spurious configurations: The ten-valued CIFAR10 construction assigns class-associated spurious configurations with probability pi,i > 0.1 and distributes the remaining probability across other configurations.For the reported setting, B = 0.5 and pi,i = 0.5, creating equal-sized Majority and Minority groups.
Non-orthogonal features: In the line-in-third-channel setting, the main dataset contains a faint invariant-image copy plus a vertical line, while the control retains only the vertical line and makes the features orthogonal.The non-orthogonal representation is modeled as (xinv, xinv + xsp), whereas the control is (xinv, xsp).
Observed failure patterns: The orthogonal control leaves OoD performance unaffected, while multidomain training is more robust than purely vertical-line training and aligned line shifts can reduce accuracy.A faint image copy can still receive enough weight to create susceptibility to shifts in the middle line.
Implicit spurious correlation: In the cats-versus-dogs construction, x1 + x2 is invariant and label-informative, whereas xdiff = x1 − x2 behaves as a shift-sensitive spurious feature.The induced spurious relation is non-visual even though color itself is independent of the label.

C.4.1 BINARY-MNIST EXPERIMENTS VALIDATING THE EFFECT OF STATISTICAL SKEWS.

The Binary-MNIST experiments isolate statistical skews by comparing balanced and duplicated datasets without geometric skews, finding greater test-time degradation under statistical imbalance.

The control dataset balances majority and minority groups, while the experimental dataset duplicates majority-aligned points to create a statistical skew without geometric skew.The control uses a 1:1 split; the duplicated dataset adds randomly selected majority points.
A 10:1 majority-to-minority ratio is created by duplicating the minority group nine times in the Binary-MNIST experimental dataset.The resulting experimental dataset contains 330k points, compared with 60k in the control dataset.
The experiments vary the training scale B between 0.1 and 1.0 and test robustness under shifts in feature scale and correlation direction.Testing includes concentration on either the minority or majority group.
The statistically skewed dataset suffers more test-time degradation than the unskewed dataset.This comparison is reported in Figure 14 under the evaluated distribution shifts.
The proposed responses to these skews are balanced max-margin training for geometric skew and weight decay or minority oversampling for statistical skew.Balanced margins downscale minority margins, while weight decay accelerates convergence and oversampling removes statistical imbalance when group information is available.
Balanced max-margin training requires knowing which datapoints belong to the minority and majority groups.This applicability boundary is explicitly noted for settings such as fairness applications.

E DEMONSTRATING SKEWS ON A NON-IMAGE-CLASSIFICATION DATASET

The obesity experiments extend the paper’s analysis beyond images, showing that geometric and statistical skews can make linear classifiers rely on a spuriously correlated public-transport feature.

The obesity demonstration is intended to show the framework’s applicability to a practically important non-image task, not to recommend interventions for high-risk decisions.The authors state that ethical and social considerations are beyond the paper’s scope.
The obesity dataset contains 16 original features and is converted into a binary task with 1,892 datapoints after excluding the middle obesity level.The data are randomly split to obtain 729 test datapoints, and the resulting 20 features are scaled to [−1, 1].
The original dataset has little public-transport correlation with the label, and its max-margin classifier has spurious component 0.008 with perfect test accuracy.The near-0.5 alignment probability indicates little original spurious correlation.
Under geometric skew, the biased training set contains 581 aligned and 10 misaligned points, producing spurious component 0.12 and test accuracies of 99.5% aligned versus 57.20% misaligned.The spurious component is about 15 times larger than in the original dataset.
The geometric-skew result is explained by the increasing ℓ2 norm required to fit larger datasets using only invariant features.Figure 16a validates this increase in the invariant feature space.
With statistical skew, gradient descent takes a long time to reduce the spurious component toward its near-zero final value, and convergence slows as skew increases.The experiment trains logistic-loss linear classifiers with SGD for up to 10k epochs.
The paper lacks datasets combining naturally embedded spurious features with explicit quantification and manipulation of those features.The authors identify this as a gap beyond the paper’s scope.