PaperlayerView exact revision on arXiv

Source-linked AI summary

The Pitfalls of Simplicity Bias in Neural Networks

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, Praneeth Netrapalli

arXiv:2006.07710v2cs.LGcs.AIstat.ML

TL;DR

The paper addresses the vague notion of simplicity bias and its inability to explain neural-network non-robustness. It designs controlled datasets with predictive features of varying simplicity, showing that extreme simplicity bias can underlie several failures and is not reliably mitigated by standard approaches.

  • Problem

    Existing simplicity-bias analyses lack a precise simplicity notion and do not simultaneously capture neural networks’ observed adversarial and distribution-shift non-robustness.

  • Method

    The paper constructs synthetic and image-based datasets with features of varying simplicity and predictive power, and studies SGD-trained networks through theory and controlled experiments.

  • Results

    Neural networks exhibit extreme simplicity bias, exclusively relying on the simplest feature while ignoring complex predictive features, which can explain poor OOD performance, adversarial vulnerability, and suboptimal generalization.

  • Takeaways & Limitations

    The proposed datasets provide a testbed for evaluating algorithms intended to avoid the pitfalls of simplicity bias.

  • Takeaways & Limitations

    The paper establishes its pitfalls in the proposed datasets, while noting that broader applicability is suggested rather than established.

Abstract

from arXiv · show

Several works have proposed Simplicity Bias (SB)---the tendency of standard training procedures such as Stochastic Gradient Descent (SGD) to find simple models---to justify why neural networks generalize well [Arpit et al. 2017, Nakkiran et al. 2019, Soudry et al. 2018]. However, the precise notion of simplicity remains vague. Furthermore, previous settings that use SB to theoretically justify why neural networks generalize well do not simultaneously capture the non-robustness of neural networks---a widely observed phenomenon in practice [Goodfellow et al. 2014, Jo and Bengio 2017]. We attempt to reconcile SB and the superior standard generalization of neural networks with the non-robustness observed in practice by designing datasets that (a) incorporate a precise notion of simplicity, (b) comprise multiple predictive features with varying levels of simplicity, and (c) capture the non-robustness of neural networks trained on real data. Through theory and empirics on these datasets, we make four observations: (i) SB of SGD and variants can be extreme: neural networks can exclusively rely on the simplest feature and remain invariant to all predictive complex features. (ii) The extreme aspect of SB could explain why seemingly benign distribution shifts and small adversarial perturbations significantly degrade model performance. (iii) Contrary to conventional wisdom, SB can also hurt generalization on the same data distribution, as SB persists even when the simplest feature has less predictive power than the more complex features. (iv) Common approaches to improve generalization and robustness---ensembles and adversarial training---can fail in mitigating SB and its pitfalls. Given the role of SB in training neural networks, we hope that the proposed datasets and methods serve as an effective testbed to evaluate novel algorithmic approaches aimed at avoiding the pitfalls of SB.

1 Introduction

The paper formalizes simplicity bias with datasets containing predictive features of varying complexity, then shows that neural networks can rely exclusively on the simplest feature, producing non-robustness and sometimes worse generalization.

  • Motivation: Prior linear analyses explain SGD’s maximum-margin solutions but cannot capture neural networks’ observed brittleness to adversarial examples and distribution shifts.Maximum-margin classifiers are inherently robust to prediction-time perturbations.
  • Dataset design: The proposed datasets combine predictive features with varying simplicity and predictive power, enabling controlled theoretical and empirical study of simplicity bias.Feature simplicity is defined through the complexity of the corresponding decision boundary.
  • Extreme simplicity bias: Neural networks can exclusively use the simplest predictive feature and remain invariant to complex predictive features, despite sufficient expressive power to combine them.This extreme simplicity bias is established theoretically for one-hidden-layer networks and empirically across architectures and optimizers.
  • Pitfalls: Extreme simplicity bias can cause small margins and reliance on spurious simple features, contributing to adversarial vulnerability and poor out-of-distribution performance.The paper also connects this behavior to universal adversarial perturbations.
  • Pitfalls: Neural networks can have unreliable confidence when complex predictive features contradict the simple feature they use.The paper frames this as a failure of confidence estimates under extreme simplicity bias.
  • Pitfalls: Extreme simplicity bias can hurt same-distribution generalization even when the simplest feature is less predictive than complex features.The paper reports this as contrary to conventional wisdom about simplicity bias.

2 Related Work

Related work links learned low-level or statistical features to non-robustness and poor out-of-distribution performance, while this paper studies feature preference through simplicity bias.

  • Feature reliance: Prior studies report that neural networks often learn spurious, low-level, or statistical features instead of semantic or high-level features, leading to poor out-of-distribution performance.Some methods improve out-of-distribution performance by encouraging higher-level features but require domain-specific knowledge.
  • Measurement: The randomized-metric criterion identifies models that rely exclusively on S and remain invariant to Sc when the corresponding randomized metrics exhibit the stated behavior.Table 1 summarizes this interpretation for randomized accuracy, AUC, and logits.
  • Simplicity-bias perspective: The paper unifies several distinctions between learned and desired features by asking why neural networks prefer one feature set over another.These distinctions include statistical versus conceptual, syntactic versus semantic, and robust versus non-robust features.

3 Preliminaries: Setup and Metrics

The paper defines classification and robustness metrics, introduces randomized tests for feature reliance, and constructs synthetic and image-based datasets with controlled feature simplicity.

  • Setup and metrics: The setup uses binary classification with a scoring function s(x) and classifier f(x), evaluating standard and δ-robust accuracy.The classifier is obtained from the scoring function through a softmax-based decision rule.
  • Setup and metrics: Randomized accuracy, AUC, and logit distributions test whether a model depends on feature subset S and is invariant to its complement Sc.S-randomization removes the label dependence of S, while Sc-randomization tests whether predictions change when Sc is randomized.
  • Synthetic datasets: The synthetic building blocks are linear, noisy linear, and k-slab distributions whose simplicity is measured by the minimum number of linear pieces needed for optimal accuracy.Linear and noisy linear blocks are simpler than k-slab blocks under this definition.
  • Synthetic datasets: The multi-dimensional datasets combine these blocks so coordinates encode different simplicity and predictive-power profiles, including LMS-k, noisy LMS-k, MS-(5,7), and MS-5.LMS-k uses a linear first coordinate and multiple k-slab coordinates; MS-5 makes all coordinates equally simple.
  • Synthetic datasets: The LSN dataset provides a theoretically tractable combination of linear, 3-slab, and non-predictive Gaussian coordinates.Its first two coordinates are predictive, while the remaining coordinates are standard Gaussians unrelated to the label.
  • Experimental design: The experiments use randomly rotated 50-dimensional synthetic data by default, with small k values chosen because SGD learns them in practice.The datasets and code are publicly available, and results are reported as robust to the exact MNIST-CIFAR class pairs.
  • Image-based datasets: MNIST-CIFAR vertically concatenates MNIST and CIFAR image blocks to mirror the simple-versus-complex structure of LMS-k.The dataset contains 50,000 training and 10,000 test images of size 3 × 64 × 32.

4 Simplicity Bias (SB) is Extreme and Leads to Non-Robustness

Theory and experiments show that SGD-trained neural networks can rely exclusively on the simplest predictive feature while ignoring complex features, producing poor OOD robustness and vulnerability to small universal adversarial perturbations.

  • 4.1 Neural networks provably exhibit Simplicity Bias (SB): On LSN, one-hidden-layer ReLU networks trained by mini-batch gradient descent provably learn only the simple linear coordinate, despite a predictive 3-slab coordinate.This reliance occurs even when the linear coordinate has a smaller margin than the slab coordinate.
  • 4.2 Simplicity Bias (SB) is Extreme in Practice: Across synthetic and image-based datasets, models exclusively use the simplest feature S and remain invariant to all complex features Sc.This pattern appears across FCNs, CNNs, GRUs, and several image-classification architectures.
  • 4.2 Simplicity Bias (SB) is Extreme in Practice: S-randomized AUC is 0.5 while Sc-randomized AUC is 1.0, showing dependence on S and invariance to Sc across varying numbers of complex features.Randomizing complex features leaves positive-logit distributions essentially unchanged, whereas randomizing S shuffles logits across positives and negatives.
  • 4.3 Extreme Simplicity Bias (SB) leads to Non-Robustness: Extreme SB can explain poor OOD performance because benign distribution shifts affecting spurious simple features can drastically degrade or nullify predictions.Randomizing complex features has negligible effect even when S and Sc have equal predictive power, so output-based shift detection may also fail.
  • 4.3 Extreme Simplicity Bias (SB) leads to Non-Robustness: 99.6% of the LMS-5, 99.9% of the MS-(5,7), and 99.3% of the MNIST-CIFAR ℓ2 UAP budget attacks the simplest feature S.These small, model-agnostic and data-agnostic perturbations can nullify model performance; shared simplicity also supports transfer across architectures.

5 Extreme Simplicity Bias (SB) can hurt Generalization

Extreme SB can hurt same-distribution generalization: SGD-trained networks may persistently select a noisy simple feature even when more complex features are more predictive and learnable.

  • 5 Extreme Simplicity Bias (SB) can hurt Generalization: Extreme SB can produce suboptimal same-distribution generalization when complex features have significantly greater predictive power than the simplest feature.The paper identifies exclusive reliance on S as the reason for this outcome.
  • 5 Extreme Simplicity Bias (SB) can hurt Generalization: 95% test accuracy is attained by FCNs of widths {100, 200, 300} and depths {1, 2}, while randomized accuracies show invariance to the 100%-predictive 7-slab features.The table reports this behavior for SGD-trained FCNs on ˆLMS-7.
  • 5 Extreme Simplicity Bias (SB) can hurt Generalization: On ˆLMS-7, the noisy linear feature S has 95% predictive power while each 7-slab feature Sc has 100% predictive power, yet SGD-trained FCNs rely only on S.Networks trained on Sc alone can attain 100% test accuracy with the stated sample size and architectures.
  • 5 Extreme Simplicity Bias (SB) can hurt Generalization: An extensive SGD hyperparameter grid search does not improve performance on ˆLMS-7, although this does not show that mitigating SB is impossible.The authors treat this as a limitation on the tested optimization approaches rather than a proof of impossibility.

6 Conclusion and Discussion

The paper uses synthetic and image-based datasets to show that extreme Simplicity Bias can explain neural-network non-robustness and can even harm generalization. Ensembles and adversarial training do not reliably remove this bias, motivating new approaches and testbeds.

  • Synthetic and image-based datasets formalize feature simplicity, support theory, and capture neural-network non-robustness observed in practice.
  • Extreme Simplicity Bias can explain poor out-of-distribution performance and adversarial vulnerability by favoring simple features over complex predictive features.
  • Vanilla ensembles mitigate Simplicity Bias only partly, continuing to rely exclusively on the simplest feature.
  • Adversarial training increases margin and robustness to some extent but leaves models invariant to complex features and below maximum possible robustness.
  • The proposed datasets provide a testbed for controlled experiments evaluating algorithms intended to avoid extreme Simplicity Bias.

Broader Impact

The authors characterize the work as foundational and do not foresee significant short-term societal consequences. They argue that understanding deep-learning phenomena is important for developing reliable systems with future societal impact.

  • The work is foundational in nature and is intended to improve understanding of neural networks.
  • The authors foresee no significant societal consequences in the short term, while linking long-term understanding to reliable deep-learning systems.

Appendices

The appendices extend the paper’s related-work discussion and provide supplementary experiment details, including additional validation of extreme Simplicity Bias and tests of mitigation methods.

  • The supplementary material adds related work and experiment details, then presents further validation of extreme Simplicity Bias.
  • The related-work discussion covers margin-based generalization bounds, adversarial attacks and robustness, and out-of-distribution examples.
  • Margin-based approaches may not fully explain neural-network generalization, and optimization may still favor small-margin classifiers using only simple features.
  • The appendices discuss adversarial-defense strategies and the brittleness of standard neural networks to adversarial attacks and benign distribution shifts.

B Experiment Details

The experiments use synthetic building-block datasets and MNIST-CIFAR variants with standard and adversarial training across several neural architectures. Dataset construction varies feature complexity, noise, margins, and class pairs to test Simplicity Bias broadly.

  • Dataset construction: Linear, noisy-linear, slab, and noisy-slab building blocks vary effective margin, width, slab count, and noise.
  • Datasets: The experiments use synthetic datasets, the LSN theoretical dataset, and MNIST-CIFAR variants combining simple MNIST with complex CIFAR-10 features.
  • Dataset construction: The synthetic suite includes one simple feature and multiple independent complex features, with default γ = 0.1, B = 1, and p = 0.1.
  • Dataset variants: MNIST-CIFAR variants change MNIST or CIFAR-10 class pairs, and Section 4 results hold across all three datasets.
  • Models and training: Models include fully connected, convolutional, sequential, and MobileNetV2 architectures evaluated under standard and adversarial training procedures.
  • Robustness checks: Additional experiments vary architectures, datasets, optimizers, activations, regularization, input dimensionality, and the number of complex features.

C.1 Effect of Model Architecture

Extreme simplicity bias persists across architectures, datasets, activations, optimizers, regularization settings, dimensions, and initialization choices. Non-random initialization can alter which feature models rely on, but training still favors the simplest available feature.

  • Architecture and dataset effects: S-randomized AUCs are approximately 0.5 while Sc-randomized AUCs are 1.00 across FCNs, CNNs, GRUs, and state-of-the-art CNNs.These results indicate exclusive reliance on the simplest feature across synthetic and image-based datasets.
  • Architecture and dataset effects: Randomizing the MNIST block drops ResNet50 AUC from 1.0 to 0.5, whereas randomizing CIFAR10 has no effect.The models therefore rely on the simpler MNIST block rather than the complex CIFAR10 block.
  • Training choices: All tested activation-function and optimizer combinations retain extreme simplicity bias, with S-randomized AUCs near 0.50 and Sc-randomized AUCs near 1.0.This includes ReLU, Leaky ReLU, PReLU, and Tanh with SGD, Adam, and RMSProp.
  • Training choices: Dropout, ℓ2 regularization, lower input dimension, and fewer complex features do not eliminate exclusive reliance on the simplest feature.The same randomized-AUC pattern persists across these settings.
  • Initialization: Kaiming or Xavier initialization, with scales from 0.1 to 10.0, does not alter extreme simplicity bias.SGD-trained FCNs continue relying on the linear component rather than 7-slab coordinates.
  • Initialization: After training on LMS-7, models initialized by interpolation rely exclusively on the linear coordinate when α ≤0.5, while larger α preserves some complex-feature dependence.Before training, the interpolated models exclusively rely on 7-slab features.

D.1 Experiment Setup in Section 5

The Section 5 experiment trains FCNs on noisy-linear-plus-complex-feature data while searching broadly over SGD settings. The setup tests whether simplicity bias can produce suboptimal generalization despite available complex predictors.

  • Dataset and models: FCNs of widths 100, 200, and 300 and depths 1 and 2 are trained with SGD on 40,000 examples from a 50-dimensional noisy LMS-7 dataset.The dataset contains one linear coordinate with 10% noise and 49 noiseless 7-slab coordinates.
  • Optimization: The hyperparameter search spans learning rate, batch size, weight decay, and momentum settings.The listed grid includes five learning rates, four batch sizes, three weight-decay values, and three momentum values.
  • Optimization: Models train for at most 10 million updates with a constant learning-rate schedule and early stopping when loss diverges or falls below 0.01.The update budget corresponds to 1,000 epochs with batch size 4 or 64,000 epochs with batch size 256.
  • Evaluation: The selected hyperparameters are those producing the best validation accuracy among 180 combinations.Table 10 reports the winning SGD settings for FCNs trained on noisy LMS-7 data.

E.1 Ensemble Methods

Ensembles help when independently trained models diversify across equally simple predictive features, but they fail when simplicity bias makes every model choose the same noisy feature. Adversarial training improves robustness somewhat without removing simplicity bias or achieving optimal robustness.

  • Ensembles: Ensembles of sizes 5 and 10 improve MS-5 generalization by approximately 15% and 20%, respectively.MS-5 contains multiple features with equal simplicity and predictive power, so independently trained models rely on different coordinates.
  • Ensembles: On noisy LMS-7 data, ensembles of sizes 3, 5, and 10 leave test accuracy at 75%.All models latch onto the same noisy linear block and therefore make the same misclassifications.
  • Adversarial training: Adversarially trained AdvMS-(5,7) FCNs do not learn maximum-margin classifiers or achieve 100% γdata-robust accuracy.Increasing width improves robustness somewhat, but large perturbation norms cause optimization-related failures.
  • Adversarial training: On MNIST-CIFAR, adversarial training significantly improves ϵ-robust accuracy without reducing standard accuracy, yet CNNs remain invariant to CIFAR10.Thus the models do not achieve the best possible ϵ-robust accuracy.

F Proof of Theorem 1

The proof analyzes mini-batch gradient descent on a stylized Linear-Slab-Noise distribution. It shows that the linear coordinate dominates learned weights, producing low test error but sacrificing margin by ignoring the predictive slab feature.

  • Data model: LSN combines a predictive singleton linear coordinate, a predictive 3-slab coordinate, and non-predictive Gaussian noise coordinates.The linear and slab blocks have supports {−1,1} and {−1,0,1}, respectively.
  • Theorem: Theorem 1 states that one-hidden-layer ReLU networks trained with mini-batch GD exclusively rely on the simple linear coordinate at the cost of margin.The result holds despite the presence of the predictive slab coordinate.
  • Theorem: After O(1) iterations, the learned network has test error at most 1/poly(d), while its hidden-unit weights satisfy the theorem’s linear-dominance estimates.The theorem assumes Gaussian initialization, a specified sample-size range, and a step size η = Ω(log d)^−1/2.
  • Proof strategy: The induction maintains active hinge loss, derives closed-form population gradients, and controls linear, slab, and noise coordinates across iterations.Lemmas 2, 4, 5, and 6 establish the loss condition, gradients, and coordinate estimates used by Lemma 1.
  • Proof strategy: The proof shows linear-coordinate weights are larger than slab- and noise-coordinate weights, while noise-gradient norms remain small.These bounds preserve the inductive conditions and support the final low-error conclusion.
Loading 2006.07710v2…