PaperlayerView exact revision on arXiv

Source-linked AI summary

Double Descent Risk and Volume Saturation Effects: A Geometric Perspective

Prasad Cheema, Mahito Sugiyama

arXiv:2006.04366v2stat.MLcs.LG

TL;DR

The paper asks why double-descent behavior can contradict traditional AIC/BIC expectations that error rises with model dimensionality. It uses model volume, motivated by MDL, Occam’s Razor, and information geometry, to analyze isotropic linear regression and statistical lattice models. The resulting decomposition of log V suggests that volume saturation and compressibility can help explain why generalization error need not keep increasing in over-parameterized regimes.

  • Problem

    AIC/BIC-style O(D) complexity suggests worsening out-of-sample error with dimensionality, yet highly parameterized models can generalize well and exhibit double descent.

  • Method

    The paper analyzes log model volume using MDL, Occam’s Razor, and Fisher-information geometry for isotropic linear regression and statistical lattice models.

  • Results

    For the studied model classes, log V can decrease with dimensionality and counteract O(D) complexity, while volume-related components help explain risk saturation.

  • Takeaways & Limitations

    Some highly parameterized models may generalize well because compressibility and saturation effects prevent test error from increasing catastrophically as dimensionality exceeds sample size.

  • Takeaways & Limitations

    The analysis is constrained by singular geometries in regimes such as D > N and is not immediately applicable to latent hierarchical lattice models.

Abstract

from arXiv · show

The appearance of the double-descent risk phenomenon has received growing interest in the machine learning and statistics community, as it challenges well-understood notions behind the U-shaped train-test curves. Motivated through Rissanen's minimum description length (MDL), Balasubramanian's Occam's Razor, and Amari's information geometry, we investigate how the logarithm of the model volume: $\log V$, works to extend intuition behind the AIC and BIC model selection criteria. We find that for the particular model classes of isotropic linear regression and statistical lattices, the $\log V$ term may be decomposed into a sum of distinct components, each of which assist in their explanations of the appearance of this phenomenon. In particular they suggest why generalization error does not necessarily continue to grow with increasing model dimensionality.

1 Introduction

The paper frames double descent as a challenge to AIC/BIC intuition that out-of-sample error rises with model dimensionality, and proposes log model volume as a geometric extension of model selection. It connects this term to MDL, Occam’s Razor, and the Fisher-information geometry underlying statistical models.

  • 1 Introduction: AIC and BIC suggest that out-of-sample error increases with model dimensionality, consistent with traditional U-shaped train-test curves.These criteria rely on asymptotic assumptions in sample size and include O(D)-type complexity terms.
  • 1 Introduction: Highly parameterized models can instead generalize well, motivating explanations for the double-descent risk phenomenon.The phenomenon has been observed across several model classes and is illustrated in Figure 1.
  • 1 Introduction: The paper studies model volume geometrically, arguing that log V can decrease with dimensionality and overpower the O(D) model-complexity term.This provides the paper’s proposed explanation for why generalization need not deteriorate in over-parameterized regimes.
  • 1.1 Model Selection and Occam’s Razor: Rissanen’s MDL and Balasubramanian’s Bayesian Occam’s-Razor formulation add a model-distinguishability term based on the Fisher information matrix.Their criteria retain an AIC/BIC-like complexity contribution alongside the integral involving det(I(θ)).
  • 1.2 Information Geometry: Information geometry identifies the Fisher information matrix as a Riemannian metric whose volume element gives the log-volume interpretation of the additional term.The paper notes that singular Fisher information matrices can invalidate classical asymptotic assumptions and the usual BIC–marginal-likelihood equivalence.

2 Double Descent Risk and the Volume of Statistical Models

This section uses the volume of statistical models to explain double-descent behavior in isotropic linear regression, including the singular and regularized regimes. It relates risk peaks, saturation, and their mitigation under regularization to model-volume growth, distinguishability, and signal-to-noise ratio.

  • 2.1 Isotropic Linear Regression: The unregularized volume is well-defined for D ≤ N but becomes degenerate for D > N because X^T X has rank N and a null space of dimension D − N.A regularized log-volume adds αI, extending the row-space geometry and making the volume well-defined in the modern regime.
  • 2.1 Isotropic Linear Regression: The original log-volume separates into a data-dependent determinant term and a noise-scaled D-ball volume that measures model distinguishability.The ball-volume component satisfies lim_D→∞ B_D(R) = 0, providing the geometric source of volume shrinkage.
  • 2.1 Isotropic Linear Regression: The MDL interpretation supports geometric intuition but does not directly make the proposed code length a generalization-error bound.The regularization increases the hypothesis-code length by the logarithm of α-scaled principal components while leaving the data-given-hypothesis term unchanged.
  • 2.1 Isotropic Linear Regression: Mean-regularized log-volume connects to channel capacity, which bounds reliable information transfer and equals the supremum of mutual information for the isotropic regression setting.Averaging over X incorporates channel-capacity effects, which are linked to input-output mutual information and generalization error.
  • 2.1 Isotropic Linear Regression: For D ≤ N, expected regularized log-volume retains the classical AIC/BIC-like O(D) contribution alongside a power-constraint term under sufficiently abundant data.The relevant assumption is N ≫ D/SNR, and the power constraint acts as strict regularization over parameter space.
  • 2.1 Isotropic Linear Regression: In D > N isotropic regression, channel capacity saturates with D, while the shrinking parameter-ball volume can counterbalance conventional complexity and prevent generalization from necessarily exploding.Theorem 3 identifies an O(D log D) complexity term, an N/2 log(SNR + 1) saturation term, and log B_D(1), which tends to −∞ as D increases.
  • 2.1 Isotropic Linear Regression: The experiments vary N = 300, 600, 900 and α = 10^-2, 10^0, 10^2 with fixed seeds, coefficient variance P = 0.25, and noise variance 1.These choices maintain a fixed SNR across runs and are intended to make the experiments straightforward and reproducible.
  • 2.1 Isotropic Linear Regression: Test risk peaks at N = D, then saturates as D increases for every α, while train-test curves converge to the same asymptotic error under fixed SNR.The interpolation peak is observed for α = 10^-2, and saturation is attributed to the limiting behavior of C as D →∞; the asymptotic dependence on SNR follows Hastie et al..
  • 2.1 Isotropic Linear Regression: Stronger regularization reduces β-norm growth and progressively weakens or eliminates double-descent risk, unlike weak regularization at the interpolation threshold.The volume V2 expands relative to noise spheres when P and ||β||2 become large, increasing out-of-sample test error; α = 10^0 and 10^2 constrain this expansion.
  • 2.1 Isotropic Linear Regression: Increasing N shifts the double-descent peak rightward, so insufficient distinguishability can make additional data worsen generalization after retraining.This interpretation follows Rissanen’s MDL account and accords with Nakkiran et al.’s observation that more data can hurt for a fixed architecture and training procedure [30].
  • 2.1 Isotropic Linear Regression: The volume-based distinguishability intuition connects double descent to jamming transitions and sphere-covering or sphere-packing arguments in statistical generalization.The paper relates these perspectives to Geiger et al. [21] and sphere-covering results.

3 Statistical Lattice Models and Model Volumes

Statistical lattice models use poset-based information geometry to analyze model volume, which decomposes into model richness and model distinguishability. The simplex constraint makes volume shrink factorially with dimensionality, producing a double-descent-like MDL pattern, while the analysis is limited for latent or singular models.

  • Geometric construction: The lattice construction uses a finite poset and η-coordinates, with the partition-function coordinate removed so the remaining metric tensor is positive definite.The original metric has a zero first row and column, while the reduced coordinates η′ = (η2, . . . , ηD) yield G′ ≻ 0.
  • Lattice volume bounds: Theorem 5 bounds lattice log V after reparameterizing η-coordinates into a Dirichlet-sampled simplex representation.The bounds use G = M^⊺M and Γ(D) = (D −1)! to make the volume integral tractable.
  • Volume decomposition: The lattice log V term separates into model richness from the metric tensor and model distinguishability from simplex volume.The distinguishability component equals 1/Γ(D) = 1/(D −1)!, so it decreases as dimensionality grows.
  • Double descent: As D increases, the MDL expression first rises and then falls because factorially shrinking simplex volume counterbalances the growing AIC O(D) complexity term.This behavior is presented as a double-descent-like pattern and illustrated with toy values in Figure 4.
  • Scope: The analysis assumes a fully observable lattice model and is not immediately applicable to restricted Boltzmann machines because latent hierarchies can produce singular geometry.Such singularity complicates global model-volume calculations based on the Fisher information matrix.
  • Asymptotic behavior: The volume of the proposed statistical lattice models tends to zero factorially as D becomes large.This limiting behavior follows from the simplex-volume factor.

4 Conclusion

The paper argues that geometric analysis of log V clarifies saturation effects in modern statistical models. For some models, high compressibility supports an Occam’s-Razor interpretation of non-catastrophic test error when dimensionality greatly exceeds sample size.

  • 4 Conclusion: Geometric analysis of log V reveals saturation effects that help explain why out-of-sample test error need not become catastrophic when dimensionality far exceeds the number of datapoints.The conclusion connects this behavior to model compressibility and an Occam’s-Razor-like principle.

A.1 Fisher Information Matrix in Linear Regression

The appendix derives the Fisher information matrix for the proposed linear regression model and uses power-constrained coefficient distributions to justify high-probability volume calculations. The construction relies on the known variance assumption and sub-Gaussian concentration.

  • Fisher information matrix: The linear regression Fisher information matrix is G(β) = E[X^⊺X] /σ^2, obtained by negating the probabilistic regression expression and taking expectation.The derivation assumes that σ^2 is known.
  • Power constraint: The volume calculation integrates over the hard power constraint β^⊺β ≤ P in R^D.The later random-coefficient formulation uses β_i ∼ N(0, P/D), whose expected squared norm is P.
  • High-probability volume calculation: The Gaussian coefficient choice follows a maximum-entropy argument under fixed mean and covariance, while sub-Gaussian concentration keeps the norm near the D-sphere radius with high probability.This concentration ensures that the D-ball volume calculations hold with high probability.

A.2 Channel Capacity Redundancy Theorems

The redundancy-capacity results connect channel capacity to min-max KL risk and, asymptotically, to the MDL generalization estimator. This establishes a relationship between channel capacity and out-of-sample test error.

  • Redundancy-capacity relation: The redundancy-capacity theorem equates maximum channel capacity with min-max KL divergence.The paper attributes this theorem to Clarke and Barron.
  • Redundancy-generalization relation: In the infinite-data limit, min-max KL risk approaches the MDL generalization estimator.This is stated as the redundancy-generalization theorem [17].
  • Implication: Together, the theorems establish an asymptotic relationship between channel capacity and out-of-sample test error through the min-max KL risk.The shared term is inf_p∈P sup_β∈B KL(p(y|β)∥p(y)).

A.3 Proof of Theorem 1

The proof derives Theorem 1’s channel-capacity expression for real Gaussian design and noise by applying Pinsker’s mutual-information identity to covariance matrices and determinant expansions.

  • A.3 Proof of Theorem 1: Pinsker’s theorem expresses the Gaussian mutual information through covariance matrices and the logarithm of a determinant.The proof identifies β with ξ and y with η, then uses their covariance matrices in the mutual-information formula.
  • A.3 Proof of Theorem 1: The proof first conditions on a fixed design matrix X before extending the calculation to random X.The fixed-X calculation is later related to the desired mutual information involving both y and X.
  • A.3 Proof of Theorem 1: A maximum-entropy Gaussian prior for β under the power constraint supplies the input covariance used in the derivation.The prior assigns equal variance across dimensions, while the noise covariance is σ^2I.
  • A.3 Proof of Theorem 1: Block covariance matrices and determinant expansion produce the channel-capacity expression for the isotropic linear-regression setting.The cross-covariance blocks are transposes of one another, enabling the determinant calculation.

A.4 Theorem 9 - Channel Capacity Bounds

The section establishes separate upper and lower channel-capacity bounds for D ≤ N and D > N, then examines their behavior across signal-to-noise ratios and dimensions.

  • A.4 Theorem 9 - Channel Capacity Bounds: Four bounds are derived by combining upper and lower bounds across the two regimes D ≤ N and D > N.The theorem states these bounds using the digamma function Ψ.
  • A.4 Theorem 9 - Channel Capacity Bounds: For D > N, the upper-bound expression includes a dimension-independent term proportional to 2 log(SNR + 1).The supplied passage states this form for D > N.
  • A.4 Theorem 9 - Channel Capacity Bounds: The bounds use determinant inequalities, monotonicity of logarithms, and the expected log-determinant of a Wishart matrix.The Wishart distribution arises because X^⊺X is Wishart distributed when the design entries are standard Gaussian.
  • A.4 Theorem 9 - Channel Capacity Bounds: Figure 5 compares the channel-capacity bounds across SNR, with the lower bound appearing tighter at high SNR but dipping near D → N.The dip is attributed to a lower-bound term that becomes negative when SNR < D/2.

A.5 Corollary 1 - Convergence of Channel Capacity Limit

The paper shows that channel-capacity bounds converge as D grows under high SNR, interprets AWGN capacity geometrically through sphere packing, and reparameterizes lattice coordinates through a simplex mapping.

  • A.5 Corollary 1 - Convergence of Channel Capacity Limit: Under high SNR and the large-argument digamma approximation, both capacity bounds converge as D → ∞ for D > N.The common limit is stated as N/2 log(SNR).
  • A.6 Sphere Packing in AWGN: For AWGN, transmitted signals are surrounded by noise spheres, and the decodeable codewords lie within an approximate radius determined by signal and noise power.Figure 6 presents this channel-capacity interpretation as sphere packing on a hypersphere.
  • B.1 Reparameterzing the Dual Geometry of Lattices: Lattice distributions use hierarchical partial-order structure and belong to the exponential family, including models such as Boltzmann machines and log-linear models.The binary lattice example is defined over {0, 1}^n and encodes higher-order interactions.
  • B.1 Reparameterzing the Dual Geometry of Lattices: The η coordinates are recursively reparameterized through simplex coordinates δ, yielding η = Zδ with an upper-triangular zeta matrix.Because the zeta matrix has determinant one, the coordinate transformation preserves the relevant Jacobian determinant.
  • B.1 Reparameterzing the Dual Geometry of Lattices: The lattice volume integral is expressed using a Dirichlet distribution over the simplex, with αd = 1 selecting uniform exploration.Bounding the volume reduces to bounding a logarithmic term and appending log Γ(D).

B.2 Proof of Theorem 5

The proof decomposes the lattice log-volume calculation using a Cholesky factorization, revealing additive components associated with richness and distinguishability.

  • B.2 Proof of Theorem 5: The positive-definite metric matrix is factorized as G = MM^⊺ using a triangular Cholesky factor.This factorization supports the subsequent decomposition of the volume expression.
  • B.2 Proof of Theorem 5: The resulting log-volume expression separates into distinct additive components of richness and distinguishability.The paper connects this decomposition to analogous results for isotropic linear regression and the stochastic perceptron unit.

B.3 Proof of Remark 2

The proof establishes that model volume vanishes as dimensionality grows, while the stochastic perceptron extends the volume decomposition into distinguishability and richness components.

  • B.3 Proof of Remark 2: As D →∞, V →0 and log V →−∞, allowing the volume term to overpower the traditional O(D) complexity contribution.This is presented as an indicator of increasing generalization performance.
  • C Stochastic Perceptron Unit: The stochastic perceptron uses Gaussian inputs, sigmoid activation, and Gaussian noise to formulate its information-geometric model.The parameterization reduces w · x to wξ with ξ ∼N(0, 1).
  • C Stochastic Perceptron Unit: For the stochastic perceptron, log V decomposes into distinguishability and richness components, with richness encoding activation derivatives, input-weight norms, and input distributions.Unlike isotropic linear regression, the richness term directly reflects model architecture.
  • C Stochastic Perceptron Unit: Bounded derivatives in sigmoid, ReLU, and tanh activations support V →0 as D →∞, although singular geometry complicates extension to deeper networks.The paper notes that these activations satisfy |f′(·)| ≤1.

C.1 Stochastic Perceptron Metric Tensor

The stochastic perceptron metric tensor is derived under Gaussian-input and Gaussian-noise assumptions, then rewritten in expectation form to support volume calculations.

  • C.1 Stochastic Perceptron Metric Tensor: The stochastic perceptron assumes x ∼N(0, I), ε ∼N(0, σ2), and a sigmoid activation, with w · x represented as wξ.This parameterization follows Amari’s formulation and expresses the input dependence through a scalar Gaussian variable.
  • C.1 Stochastic Perceptron Metric Tensor: Theorem 11 provides the metric tensor for the stochastic perceptron under the stated parameterization.The theorem is identified as the stochastic perceptron metric tensor.
  • C.1 Stochastic Perceptron Metric Tensor: The metric tensor is condensed into expectation form by taking expectations over the Gaussian variable ξ.The expectation formulation incorporates derivative and noise terms, using y −f(w · x) = ε and E[ε2] = σ2.
  • C.1 Stochastic Perceptron Metric Tensor: The resulting volume formula is arranged to expose separate richness and distinguishability contributions.This form makes the geometric roles of the two components explicit for subsequent analysis.

C.2 Proof of Theorem 10

The proof extends Amari’s volume expression by applying determinant and radial-integration steps that reveal its richness and distinguishability structure.

  • C.2 Proof of Theorem 10: The paper extends Amari’s previously unproved volume expression by providing a proof that makes richness and distinguishability terms explicit.The extension is presented for a generic Gaussian noise term.
  • C.2 Proof of Theorem 10: The matrix-determinant lemma is applied to the perceptron metric tensor as the central algebraic step in the proof.This connects the metric tensor to the determinant underlying the volume expression.
  • C.2 Proof of Theorem 10: Integration over a D-ball introduces the radial volume element BD(1)wD−1dw.This radial form supports separating the geometric contribution from the remaining terms.
  • C.2 Proof of Theorem 10: The coefficients c1(w) and c2(w) are expressed through expectations involving the activation derivative and Gaussian variable.These coefficients supply the terms used in the volume decomposition.
Loading 2006.04366v2…