PaperlayerView exact revision on arXiv

Source-linked AI summary

The Low-Rank Simplicity Bias in Deep Networks

Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, Phillip Isola

arXiv:2103.10427v4cs.LGcs.CV

TL;DR

The paper asks why highly over-parameterized deep networks can generalize rather than overfit. It empirically studies whether depth biases linear and nonlinear networks toward low-effective-rank embeddings, finding this bias at initialization and after training and using it to improve CIFAR and ImageNet generalization without increasing modeling capacity.

  • Problem

    The paper addresses why heavily over-parameterized deep networks often generalize instead of overfitting, extending prior analysis beyond mostly linear or under-explored nonlinear settings.

  • Method

    The authors empirically measure effective rank in feature embeddings and Gram matrices across finite-width linear and nonlinear networks, depths, initializations, optimizers, and training settings.

  • Results

    Deep networks show a low-rank embedding bias at initialization and after training, across common optimizers, and depth-based linear over-parameterization improves generalization on CIFAR and ImageNet without increasing modeling capacity.

  • Takeaways & Limitations

    The observations support treating depth as an inductive bias toward low-effective-rank solutions and as a possible route to better generalization when such solutions are appropriate.

  • Takeaways & Limitations

    The preliminary theoretical analysis relies on assumptions including infinite-width networks and weight distributions, while finite-width results are demonstrated empirically rather than proved.

Abstract

from arXiv · show

Modern deep neural networks are highly over-parameterized compared to the data on which they are trained, yet they often generalize remarkably well. A flurry of recent work has asked: why do deep networks not overfit to their training data? In this work, we make a series of empirical observations that investigate and extend the hypothesis that deeper networks are inductively biased to find solutions with lower effective rank embeddings. We conjecture that this bias exists because the volume of functions that maps to low effective rank embedding increases with depth. We show empirically that our claim holds true on finite width linear and non-linear models on practical learning paradigms and show that on natural data, these are often the solutions that generalize well. We then show that the simplicity bias exists at both initialization and after training and is resilient to hyper-parameters and learning methods. We further demonstrate how linear over-parameterization of deep non-linear models can be used to induce low-rank bias, improving generalization performance on CIFAR and ImageNet without changing the modeling capacity.

1 Introduction

The paper investigates why over-parameterized deep networks can generalize by studying their apparent bias toward low-effective-rank embeddings. Empirical results suggest this bias appears at initialization and after training, across optimizers, and can improve generalization without increasing modeling capacity.

  • Motivation: Deep networks are over-parameterized yet often avoid overfitting, motivating analysis of how depth influences the solutions they find.The paper contrasts deep networks’ expressivity and parameter counts with their observed generalization.
  • Motivation: Prior work has mainly studied over-parameterization in linear models or treated its implicit regularization in nonlinear networks as an under-explored observation.This work extends the analysis to empirical experiments on both linear and nonlinear networks under practical learning paradigms.
  • Core observations: Deep networks exhibit an inductive bias toward lower-rank embeddings, both at initialization and after gradient-descent training.The paper presents this as its central empirical observation.
  • Core observations: The low-rank bias persists across common optimizers, including methods that do not use gradient descent, and converged rank depends substantially on model depth despite initialization rank.This supports a depth-dependent bias rather than an effect restricted to one optimizer or initialization.
  • Conjecture: The authors conjecture that depth favors low-effective-rank embeddings because the volume of functions mapping to such embeddings increases with depth.This is presented as a conjecture motivated by the empirical observations.
  • Generalization: Linear over-parameterization by depth improves generalization on CIFAR and ImageNet without increasing modeling capacity.The paper uses this observation to connect the low-rank bias to practical performance.

2 Preliminaries

The paper defines effective rank as a continuous measure of singular-value spread and studies it through feature embeddings and their Gram matrices. Its setup compares linear and nonlinear networks while controlling for data-dependent matrix dimensionality and training regime.

  • Network parameterization: A linear network can be over-parameterized by factorizing one effective weight matrix into successive weight matrices without changing the represented function class.The expanded weights are distinguished from the collapsed end-to-end weights.
  • Network parameterization: Nonlinear networks interleave activation functions between weight matrices and become more expressive as layers are added.This distinguishes nonlinear depth from linear factorization, whose effective function class can remain unchanged.
  • Effective rank: Effective rank ρ is defined as the Shannon entropy of normalized singular values, providing a continuous measure of matrix rank.It is maximized when singular-value magnitudes are equal and minimized when one singular value dominates.
  • Initialization bias: At initialization, adding layers shifts the effective-rank distribution toward lower values, with the experiment repeated under normal and uniform parameter distributions.The distributions are estimated from Gram matrices computed using random network parameter samples.
  • Embedding maps: The embedding analysis computes effective rank on a Gram matrix formed from intermediate feature representations.The Gram matrix is indexed by data samples, and its entries use a distance kernel between feature vectors.
  • Experimental regime: For nonlinear networks, comparisons are made at zero training error, while the Gram-matrix dimension depends on data samples rather than model parameters.The regression setup uses an under-determined regime with fewer training examples than model parameters, leaving multiple minimizing solutions.

3 The parameterization bias of depth

The paper argues that depth biases networks toward lower effective-rank embeddings through parameterization, rather than optimizer choice alone. Empirical results show this bias at initialization and after training across linear and nonlinear models.

  • 3 The parameterization bias of depth: Deeper networks have a greater proportion of parameter space mapping inputs to lower-rank embeddings, making simpler embeddings more likely outcomes.This is the paper’s central conjecture about parameterization bias.
  • 3 The parameterization bias of depth: Randomly initialized deep networks produce Gram matrices with lower effective rank, across both linear and nonlinear models.The authors estimate effective-rank distributions from thousands of randomly sampled parameter settings.
  • 3 The parameterization bias of depth: After gradient-descent training, deeper models still find lower effective-rank solutions, even when they achieve zero training loss.The converged rank shifts toward the task rank but remains constrained by network depth.
  • 3 The parameterization bias of depth: Deeper nonlinear networks show decreasing Gram-matrix effective rank and increasingly block-structured embeddings across several activation functions.The visualized activations include ReLU, Tanh, GeLU, and sinusoidal SIREN functions.
  • 3 The parameterization bias of depth: The low-rank embedding bias persists across common optimizers, although the particular converged solution depends on the optimizer.The experiments compare multiple optimization methods while holding functional expressivity fixed.
  • 3 The parameterization bias of depth: The bias is present at initialization and convergence, and deeper networks remain lower-rank when initialization distributions are matched.The authors attribute initialization differences partly to products of normally distributed weights, while matched-initialization experiments preserve the depth effect.
  • 3 The parameterization bias of depth: For linear networks, increasing depth monotonically decreases effective weight rank in the infinite-width limit under the theorem’s distributional assumptions.The theoretical result assumes normally distributed parameters and width tending to infinity.

4 Over-parameterization as a regularizer

The paper uses linear over-parameterization inside nonlinear networks to induce low-rank embeddings without increasing functional capacity. This improves generalization on image classification tasks, with gains depending on expansion and model depth.

  • 4 Over-parameterization as a regularizer: Linear over-parameterization induces a low-rank bias without increasing modeling capacity and improves generalization on classification tasks.The approach modifies linear components inside nonlinear networks while preserving functional equivalence.
  • 4.1 Image classification with over-parameterization: A CIFAR100 CNN with linear over-parameterization learns lower-rank embeddings, exhibits less overfitting, and generalizes better than its functionally equivalent original model.Training dynamics track normalized Gram-matrix singular values for both models.
  • 4.1 Image classification with over-parameterization: On CIFAR100, expansion factor 4 yields a +6.3 test-accuracy gain, while CIFAR10 gains +2.8 under the reported best-configuration comparison.The optimal expansion factor depends on the original network depth.
  • 4.1 Image classification with over-parameterization: On ImageNet, over-parameterization produces moderate gains for AlexNet and smaller gains for deeper ResNet architectures.The paper reports this pattern for AlexNet, ResNet10, and ResNet18.
  • 4.1 Image classification with over-parameterization: The best expansion is a sweet spot rather than an unlimited increase, and improvements are generally observed for models with fewer than 20 layers.The reported ablation identifies 4× as the best expansion for the tested CNNs.
  • 4 Over-parameterization as a regularizer: Linear over-parameterization consistently outperforms explicit low-rank regularizers while models retain the same effective number of parameters.The explicit comparisons include norm-based penalties and pseudo-measures of rank.

5 Discussion

The discussion frames linear-layer stacking as a reparameterization that preserves hypothesis space while changing which solutions models find. On ImageNet, linear over-parameterization improves generalization at equal effective parameter count, with benefits eventually plateauing.

  • Reparameterization: Stacking linear layers reparameterizes neural networks without changing their hypothesis space, potentially altering the solutions optimization reaches.The paper distinguishes searching for better parameterizations from searching for larger hypothesis spaces.
  • Reparameterization: The paper presents parameterization choice as a distinct design factor that can determine which solution a model converges to even when the represented function class is unchanged.This separates parameterization effects from changes in modeling capacity.
  • Empirical implication: Linear over-parameterization improves ImageNet generalization while preserving the original architectures’ effective parameter count.The benefit plateaus with greater depth, and no noticeable improvement appears starting from ResNet34.

6 Related works

Related work connects low-rank bias to linear networks, simplicity bias, and complexity measures, while this paper emphasizes broader empirical scope and a limitation on when the bias helps.

  • Linear and low-rank bias: Prior work studies low-rank or low-nuclear-norm solutions in deep linear networks, often under matrix-factorization or least-squares settings.These analyses commonly involve gradient descent and, in some cases, isometric assumptions.
  • Simplicity bias: Other work characterizes simplicity through max-margin solutions, low-complexity functions, or spectral properties of neural tangent and conjugate kernels.These perspectives use distinct measures of simplicity across linear and nonlinear models.
  • Complexity measures: Effective rank measures matrix effective dimensionality from the entropy of normalized singular values, whereas nuclear norm is not invariant to scaling.The paper therefore treats effective rank as a spectral alternative to nuclear norm.
  • Scope and limitation: The paper does not claim that over-parameterization always improves generalization: excessive depth can over-regularize, and low-rank bias can hurt when the target function is not low-rank.The bias is described as beneficial when the underlying function is low-rank, as often assumed for natural data.
  • Experimental scope: The experiments either assume zero training error or compare models with the same modeling capacity.This assumption defines the setting for the paper’s empirical comparisons.

Q: What is the contribution of work?

The paper extends low-rank and simplicity-bias analysis to finite nonlinear networks and practical learning settings, arguing that the bias is present before and after training and is not explained solely by gradient descent. Its empirical comparisons include spectral distributions, Gram matrices, and equal-capacity or equal-training-error models.

  • Contribution: The paper targets a gap in prior analyses that often rely on linearity, infinite width, isometry, gradient flow, or specialized learning paradigms.It investigates low-rank bias empirically in finite linear and nonlinear networks.
  • Contribution: The work extends simplicity-bias observations to finite-width linear and nonlinear networks without isometric assumptions and in practical problem settings.This broadens earlier analyses beyond more restrictive theoretical regimes.
  • Contribution: Low effective-rank bias appears both before and after training, indicating that optimization alone is not the sole explanation.The paper reports the parameterization bias with and without optimization and after convergence.
  • Contribution: The implicit bias persists beyond gradient descent, whereas prior theory mainly addressed models optimized with gradient descent.The paper presents this as an extension of the scope of implicit-bias analysis.
  • Generalization: Models with different depths can exhibit different generalization properties even when they have equal capacity and achieve zero training error.This comparison separates the observed effect from increased modeling capacity or training-error differences.
  • Practical evidence: Classification experiments on CIFAR10 and ImageNet show low effective-rank bias in practical learning paradigms.The paper also reports improved generalization from linear over-parameterization without increased effective capacity.
  • Novelty: The paper emphasizes that deeper models concentrate spectral distributions rather than merely reducing discrete matrix rank through multiplication.This effective-rank behavior is reported at initialization and convergence.
  • Spectral evidence: Finite-matrix singular-value distributions follow theoretical distributions, while Gram-matrix spectra become sharper and indicate lower effective rank with increased depth.The same depth-wise spectral trend is observed empirically for weights and Gram matrices, including after least-squares training.

D Comparisons of rank measures and kernel distance functions

The paper compares rank surrogates and finds effective rank closely tracks thresholded rank without requiring a subjective threshold. Across task ranks and kernel choices, depth-dependent generalization patterns remain consistent.

  • Rank measures: Effective rank most closely matches thresholded rank while avoiding threshold rank’s sensitivity to the chosen threshold.Threshold rank can change drastically with τ, whereas effective rank was selected to avoid this issue.
  • Rank measures: Stable rank is efficient to approximate but can collapse toward values near 1 when neural-network singular values have long, exaggerated tails.This limits its usefulness for making informative observations about rank.
  • Rank measures: Nuclear norm is unnormalized, so its magnitude can change without changing the output distribution.It reflects singular-value magnitude rather than relative ratios.
  • Least-squares ablation: High task rank favors shallower models, whereas low task rank favors deeper models, even when models span the same functions.In medium- and low-rank settings, models can reach zero training error while retaining different test losses.
  • Kernel distance functions: Linear, cosine, and correlation kernels exhibit the same low-rank simplicity-bias behavior.Cosine kernels were used in the main analysis, while linear and correlation kernels showed the same pattern.

E Singular value dynamics of weights

During training, singular-value distributions sharpen early, while effective rank drops rapidly and then slowly rebounds. The dynamics are tracked both layerwise and through Conv4 overlays.

  • Visualization: Figure 14 tracks individual singular values over time using unnormalized and largest-singular-value-scaled views.Cross-sections show distributions at selected epochs, with 24 singular values uniformly subsampled for visualization.
  • Layerwise dynamics: Normalized singular-value distributions sharpen early in CIFAR100 training but change little afterward.Unnormalized singular-value norms increase throughout training for most layers, except the final classification layer in the over-parameterized model.
  • Effective-rank trajectory: Conv4 effective rank rapidly decreases early in training and then increases slightly over the remaining epochs.The overlay reveals a non-monotonic trajectory rather than a persistent decline.

F Training details and model architecture

The experiments use tuned SGD-based training across least-squares and image-classification settings, with initialization and learning-rate choices adjusted for depth. Very deep least-squares models remained difficult to fit despite extensive tuning.

  • Optimization sensitivity: Deeper models require more learning-rate tuning, especially in least-squares experiments and at expansion factors of 8× or more.Typical 2× or 4× expansions require less tuning in the reported settings.
  • Image classification: Image-classification models use SGD with momentum 0.9, tuned initial learning rates, step decay, and standard crop and flip augmentation.CIFAR models are trained for 180 epochs with learning-rate drops at epochs 90 and 150.
  • Model architecture: Linear over-parameterization uses scaled-normal Kaiming initialization to preserve output variance across layer expansions.The authors report this variance control as critical for stabilizing training.
  • Least-squares training: Very deep least-squares networks failed to reach zero training error despite sweeps over learning rates, momentum, schedulers, and common optimization techniques.For depth 16 with task rank 24, the best configuration used momentum 0.9 and decay-on-plateau scheduling.

H Proof of Theorem 1

The paper analyzes how depth changes effective-rank behavior through asymptotic random-matrix calculations, empirical rank landscapes, and extensions to residual and linear-network dynamics. The theory proves decreasing effective rank with depth asymptotically, while experiments examine finite-width behavior and architectural mechanisms.

  • Proof of Theorem 1: The theorem relies on an asymptotic singular-value density for products of Gaussian random matrices.The density parameterizes the singular-value spectrum of a depth-L matrix product.
  • Proof of Theorem 1: The proof treats depth L continuously and shows effective rank ρ(W; L) decreases with L, implying a decreasing sequence at integer depths.The derivation differentiates the effective-rank integral using the Leibniz rule and establishes a negative derivative.
  • Proof scope: The proof is asymptotic as dim(W) →∞, while finite-dimensional validity is supported empirically rather than proved.The paper explicitly notes that it does not provide a finite-case proof.
  • Extension to residual connections: Residual networks plateau in effective rank after eight layers instead of exhibiting the monotonic rank contraction seen in linear feed-forward networks.The paper relates this behavior to identity preservation or reduced effective depth, with or without batch normalization.
  • Least-squares learning dynamics: Over-parameterization changes linear-network learning dynamics by introducing layer-dependent left and right preconditioning of the end-to-end gradient.The effective update rule is derived for two-layer and general d-layer linear networks.
  • Rank landscapes: Over-parameterized linear and nonlinear models show more parameter volume associated with lower-rank effective weights or Gram matrices.The rank landscapes compare single- versus two-layer linear models and shallow versus deeper ReLU constructions.

L Relationship between weight and embeddings

Effective rank in deep linear networks is closely aligned between end-to-end weights and Gram-matrix embeddings, while depth influences both ranks.

  • The effective ranks of weights and Gram matrices have an almost one-to-one relationship in deep linear models.
  • The relationship is observed for random deep linear networks applied to random subsets of MNIST.
  • The number of layers dictates the rank of both the embedding and the weights.

M Noisy linear regression with least-squares

Noisy least-squares experiments test whether depth preserves its low-rank simplicity bias under corrupted observations. Deeper networks generally avoid fitting noise and achieve better or comparable test performance than shallower networks.

  • In noisy regression, observation noise makes a low-intrinsic-dimensional relationship appear full-rank.The setup compares static noise sampled once with stochastic noise resampled every iteration.
  • For static noise, shallower networks overfit the noise, whereas deeper networks cannot and achieve better test performance.Shallower solutions also have much higher effective rank, consistent with depth regularizing noise overfitting.
  • For stochastic noise, deeper networks find even lower effective-rank solutions than in the noiseless setting.Deeper networks perform on par or better than their noiseless counterparts, while shallower networks perform worse.
  • Depth preserves the simplicity bias under both static and stochastic additive noise.The experiments vary noise levels while training networks of different depths with SGD and a learning-rate scheduler.
  • The authors report that depth improves generalization by underfitting noise in the data.They suggest this may explain generalization under weak supervision and corrupted labels.
Loading 2103.10427v4…