PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Expansion: A Unified Mechanism for How Deep Neural Networks Generalize

Chashi Mahiul Islam, Samuel Jacob Chacko, Mao Nishino, Canlin Zhang, Xiuwen Liu

arXiv:2610.23716v1cs.LGcs.AI

TL;DR

Overparameterized neural networks generalize well, but existing global theories offer limited explanation of individual test classifications and how generalization emerges during training. The paper introduces Neural Expansion, measuring directional stability around training samples with Generalization Intervals. It reports that ray coverage helps account for classifications across standard and challenging settings, including random-label learning, while noting computational and task-scope limitations.

  • Problem

    Deep networks generalize despite extreme overparameterization, while global theories leave open why particular test inputs receive particular classifications.

    Source: §1 · p. 1

  • Method

    Neural Expansion uses Generalization Intervals to quantify direction-specific prediction stability and ray coverage around training samples.

    Source: §1 · p. 2, §3 · p. 3, §4 · p. 5

  • Results

    Ray coverage appears across conditions, including random-label training, with 17%–24% coverage under random labels versus >80% on standard datasets.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors interpret standard generalization, memorization of random labels, misclassification, and adversarial susceptibility through a common directional-geometry account.

    Source: §6 · p. 9, §7 · p. 10

    GI computation remains expensive for high-dimensional inputs, and extending the classification-focused framework to regression, generation, or structured prediction requires reformulating GIs.

    Source: §6 · p. 10

Abstract

from arXiv · show

Deep neural networks demonstrate remarkable generalization despite being highly over-parameterized, challenging traditional statistical learning theories. While several theoretical paradigms such as margin-based bounds, PAC-Bayesian analyses, and neural tangent kernels provide global explanations, they often lack mechanistic insight into how generalization emerges from training dynamics. We introduce Neural Expansion, a unified geometric framework that characterizes generalization through the expansion of directionally-selective ray coverage in the input space. Using Generalization Intervals (GIs), we quantify direction-specific regions around training samples where the model's predictions remain stable. Our extensive empirical studies across diverse architectures and datasets reveal that training leads to progressive expansion of ray coverage, particularly along directions with low Jacobian singular values. This mechanism helps explain a large fraction of correctly classified test inputs, adversarial examples, and even mislabeled or out-of-distribution samples through the geometry of ray coverage rather than instance-level memorization. By connecting input-space curvature to predictive behavior, our framework provides a mechanistic foundation that complements existing global theories of deep learning.

1 Introduction

The paper frames deep networks’ generalization despite extreme overparameterization as a challenge for theories that do not explain individual test classifications. It introduces Neural Expansion, using direction-specific stability around training samples to explain classification through ray coverage.

1 Introduction

  • Despite having far more parameters than training examples, deep networks generalize well, leaving open why a specific test input receives a specific classification.The introduction contrasts this challenge with classical learning theory and global theoretical frameworks.Source: §1 · p. 1
  • Neural Expansion uses Generalization Intervals (GIs) to measure direction-specific regions around training samples where predictions remain invariant.The framework describes training as progressively expanding stable ray coverage, especially in low-Jacobian-singular-value directions.Source: §1 · p. 2
  • The paper argues that test classifications occur when test samples fall within training-sample ray coverage, rather than through instance-level memorization.Source: §1 · p. 2
  • The authors interpret standard generalization, adversarial vulnerability, long-tailed misclassification, and random-label learning through the directional structure of GIs.Source: §1 · p. 2
  • The framework links Jacobian singular values and directional stability to sample-level classification behavior.Source: §1 · p. 2

2 Background and Related Work

The background contrasts global complexity and training-dynamics theories with the paper’s local, sample-level geometric account. It situates Generalization Intervals as a way to describe directional prediction stability around training samples.

2 Background and Related Work

  • The paper frames its question against the challenge of explaining why overparameterized networks generalize, alongside representation-disentanglement hypotheses.Source: §2 · p. 2
  • VC dimension, Rademacher complexity, margin bounds, PAC-Bayesian analyses, and compression offer global insights, while GIs describe stable directional regions around samples.Source: §2 · p. 2
  • Unlike NTK approaches centered on dynamics near initialization, the paper quantifies post-training directional stability through Jacobians in realistic regimes.Source: §2 · p. 2
  • Neural Collapse describes feature and classifier alignment near convergence [Papyan et al., 2020; Han et al., 2021; Wu and Papyan, 2024], whereas the GI framework focuses on stable, low-curvature regions.Source: §2 · pp. 2–3
  • The paper presents Jacobian curvature as a geometric account of simplicity bias, benign overfitting, and generalization associated with flat loss regions.Source: §2 · p. 3

3 A Theoretical Framework for Neural Network Generalization

The framework defines directional prediction stability using Generalization Intervals and relates it to Jacobian sensitivity, margins, loss sharpness, and adversarial robustness. Its central theoretical account is that directional stability depends on both margin and sensitivity, with low-sensitivity directions yielding especially large intervals.

3 A Theoretical Framework for Neural Network Generalization

  • A Generalization Interval (GI) measures how far a classifier retains its class along a chosen direction; ray coverage collects points within their respective GI bounds.GIs are direction-dependent, so one sample can have large intervals in some directions and small ones in others.Source: §3 · p. 3

3.1 Jacobian Analysis and GI Formation

  • In trained networks, most representation-Jacobian singular values are near zero, predicting large GIs in most directions.For CIFAR-10 inputs, typically only 10–15 singular values are significantly nonzero, while over 99.5% of the spectrum is effectively zero.Source: §3.1 · p. 3

3.2 Linear Approximation of the GI and the Relation to the Margin

  • Under the stated linear approximation and residual bound, a larger sample margin indicates a larger GI, while greater directional sensitivity reduces its lower bound.The authors report that the linear approximation holds well empirically and account for nonlinear effects through the residual bound.Source: §3.2 · p. 4

3.3 GI Strictly Refines Spectral Margin Along Singular Directions

  • As a Jacobian singular value approaches zero with margin above the residual bound, the GI lower bound grows without bound, unlike the direction-agnostic spectral margin bound.GI separates sensitive and invariant directions, providing information where spectral margin theory is silent.Source: §3.3 · p. 4

3.4 Jacobian control of the input-loss gradient

  • The top Jacobian singular value controls input-space sharpness; low-curvature directions correspond to large GIs, while high-curvature directions correspond to small GIs.The paper also reports that the top right singular vector closely approximates the unconstrained steepest-ascent direction.Source: §3.4 · pp. 4–5

3.5 GI Quantifies the Adversarial Robustness Radius

  • The minimum GI over unit directions lower-bounds the norm of a successful class-changing perturbation, yielding an adversarial-robustness bound.The paper further expresses a bound using the margin, classifier norm, and largest Jacobian singular value.Source: §3.5 · p. 5

4 Experimental Setup

The study evaluates Generalization Intervals (GIs) across datasets and model families, using Jacobian-derived directions to analyze directional prediction stability and ray coverage.

4 Experimental Setup

  • Experiments span MNIST, CIFAR-10, ImageNette, random-label and long-tailed variants, and MLP, convolutional, and Vision Transformer architectures.Models are trained with SGD until convergence.Source: §4 · p. 5
  • GIs quantify directional prediction stability in input space and are computed along right singular vectors of the penultimate-layer Jacobian.A step-wise search is combined with ray-coverage measurement toward training samples and linear-approximation evaluation.Source: §4 · p. 5

5 Results

Across training conditions, ray coverage expands directionally and accompanies classification behavior. Interventions that restrict coverage reduce test accuracy, while analyses across ordinary and challenging settings associate predictions with class-level coverage.

5.1 GI Behavior Across Model Types and Training Scenarios

  • Across architectures, training scenarios, and datasets, GI analysis reveals consistent patterns, with RSVs computed from the input-to-penultimate-layer Jacobian.The default experiments use an MLP unless otherwise noted.Source: §5.1 · p. 6

5.1.1 Temporal Evolution of Generalization Intervals During Training

  • During MNIST-MLP training, mean GI and the share exceeding 1.0 rise steadily with test accuracy (Pearson r = 0.99), while RSV directions show larger, faster growth than random directions.Figure 1 tracks positive same-class GI growth and average GI along RSV and random directions over 10 epochs.Source: §5.1.1 · p. 6, Figure 1 · p. 6

5.1.2 GI Expansion in Early vs. Later Training

  • After training, average GI across RSVs rises from about 7 to 11.5, with growth concentrated in high-index, low-curvature directions and decreases in low-index directions.The comparison is between epoch 1 and epoch 50; the remaining RSV directions are null because their singular values are zero.Source: §5.1.2 · p. 6

5.1.3 Linear Approximation Along Directions with Different Singular Values

  • After training, local linearity improves monotonically with RSV index, reaching α ≈1.2 in low-curvature directions, and the longest-GI directions align with the most linear behavior.Figure 2 compares GI magnitudes and linear-approximation quality before and after training; the reported spectrum transition is around indices 300–400.Source: §5.1.3 · pp. 6–7, Figure 2 · p. 7

5.2 Effect of Ray Coverage Interventions on Generalization

  • Restricting ray coverage lowers test accuracy by 12.2 percentage points on MNIST-MLP and 7.4 points on Fashion-MNIST-CNN, while expansion raises accuracy by 1.7 and 1.1 points, respectively.The reported intervention effects are statistically significant; Table 1 reports final-epoch results averaged over three seeds.Source: Table 1 · p. 7, §5.2 · p. 7

5.3 Ray Coverage as a Mechanism for Test-Time Classification

  • Ray coverage provides a geometric account of test-time classification across standard, random-label, long-tailed, and adversarial settings.The analysis treats GIs as a mechanism for classification decisions, not only as a post-hoc measure.Source: §5.3 · p. 7

5.3.1 Test Sample Classification via Ray Coverage

  • Correctly classified and misclassified test samples show substantial ray coverage across datasets and architectures, consistent with predictions arising from the predicted class’s coverage rather than instance-level memorization.Figure 3 plots the percentage of same-class GIs exceeding 1.0; directional coverage uses a threshold of 1% of training samples.Source: §5.3.1 · p. 8, Figure 3 · p. 8

5.3.2 Neural Expansion as a Unified Framework for Challenging Cases

  • Random-label models form smaller, noisier same-label coverage zones; under long-tailed learning, low-frequency samples are absorbed into frequent-class coverage, while adversarial examples often remain in or shift between class coverage zones.Correctly classified ImageNette-ViTB adversarial examples show low coverage; misclassified examples could not be measured when they left the evaluated 10-class subset for the broader 1000-class ImageNet space.Source: §5.3.2 · pp. 8–9, Figure 4 · p. 8

6 Discussion

The discussion frames ray coverage as a shared mechanism for generalization and memorization, while distinguishing the wider, class-structured corridors of generalizing networks. The framework offers a sample-level account alongside global theories, but computational and theoretical limits remain.

6 Discussion

  • Random-label models have 17%–24% coverage versus >80% on standard datasets, while generalizing networks form wider, overlapping corridors aligned with low-curvature directions.The discussion presents ray coverage as operating in both memorization and generalization, with semantic supervision associated with denser, more structured corridors.Source: §6 · p. 9
  • The framework tracks sample-level geometry throughout training, complementing global theories that do not explain why a particular test point receives its class.The discussion contrasts this scope with PAC-Bayesian and margin bounds, and with NTK and Neural Collapse accounts of other training phases.Source: §6 · pp. 9–10
  • GI computation is expensive for high-dimensional inputs, and extending the classification-focused framework to regression, generation, or structured prediction requires reformulating GIs.A rigorous proof that gradient descent anisotropically restructures individual Jacobian singular values remains open.Source: §6 · p. 10

7 Conclusion

Neural Expansion frames generalization as stable, direction-sensitive pathways around training samples, with high-curvature directions shaping class boundaries and low-curvature directions forming invariance corridors. The appendices describe the datasets, distribution variants, model families, and training protocols used in the empirical analysis.

7 Conclusion

  • Neural Expansion unifies standard generalization, random-label learning, long-tailed errors, and adversarial susceptibility through directionally sensitive invariance pathways.The conclusion describes high-curvature directions as shaping class boundaries and low-curvature directions as forming corridors that link same-class samples.Source: §7 · p. 10

A.1 Datasets and Variants

  • The experiments use MNIST, CIFAR-10, and ImageNette, alongside random-label and long-tailed variants.The variants include RANDOM-10 and RANDOM-100, plus MNIST-LT1 and MNIST-LT2 with imbalance ratios of 6.5:1 and 78.6:1.Source: §A.1 · p. 13

A.2 Model Architectures

  • The study evaluates MLPs, CNNs, and Vision Transformers across the benchmark datasets.The CNN family includes custom models and pretrained ResNet-20, ResNet-50, and VGG-16; ImageNette uses ViT-B/16 and DeiT-S.Source: §A.2 · p. 13

A.3 Training Protocols

  • Custom models are trained with SGD to convergence using dataset-specific schedules and hyperparameters.The protocols specify 30 epochs for MNIST and ImageNette and 50 for CIFAR-10.Source: §A.3 · p. 13

A.4 Analysis Methods

The analysis estimates directional prediction stability by searching along sample-pair directions and Jacobian singular vectors, then measures how often test samples fall within predicted-class training-sample ray coverage.

A.4 Analysis Methods

  • Generalization Intervals are computed by stepping in both directions along each sample-pair vector until a decision boundary or the maximum search distance is reached.The unnormalized direction makes ε = 1 reach the second sample, so GI ≥ 1 has a consistent interpretation; the reported batch sizes are 6,000, 2,000, and 100 pairs for MNIST, CIFAR-10, and ImageNette.Source: §A.4 · p. 13
  • The Jacobian of penultimate-layer features with respect to the input is decomposed by SVD, and linear-approximation quality is measured along its right singular vectors.The approximation is assessed by cosine similarity between the network output and its first-order linear prediction.Source: §A.4 · p. 13
  • Ray coverage is measured as the percentage of predicted-class training samples whose GI toward a test sample exceeds 1.0.Figure 6 presents epoch-wise accuracy curves across models and datasets for vanilla, long-tailed, and random-label variations; Table 2 reports compute resources and runtimes for the coverage experiments.Source: §A.4 · p. 14, Figure 6 · p. 14, Table 2 · p. 14

B GI Evolution Experimental Details

During MNIST MLP training, GIs and ray coverage increase, with larger and faster expansion along Jacobian right singular vectors than along random directions. The directional divergence becomes especially pronounced after the early training epochs.

B GI Evolution Experimental Details

  • From epoch 0 to 10, mean train-pair GI rose from 0.76 to 1.24 and the share above 1.0 rose from 39% to 82%, alongside test accuracy increasing from 86.2% to 91.7%.The experiment tracks randomly selected same-class training pairs in an MNIST MLP.Source: B GI Evolution Experimental Details · p. 14
  • By epoch 10, mean GI along RSV directions reached 1.18 versus 0.82 along random directions, with 78% versus 34% exceeding 1.0.Both direction types used the same 56.0 scaling factor for comparison.Source: B GI Evolution Experimental Details · pp. 14–15
  • RSV relative GI growth reached 1220% by epoch 10, compared with 940% for random directions.The curves grew similarly through epoch 3, then RSV growth accelerated; Figure 7 depicts the comparison.Source: B GI Evolution Experimental Details · p. 15, Figure 7 · p. 15
  • Table 3 separates GI statistics for correctly and incorrectly classified training pairs, distinguishing the mean over all pairs from the mean over pairs with nonzero GI.The latter is labeled Avg GI; the all-pair mean includes GI = 0 boundary cases.Source: B GI Evolution Experimental Details · p. 15, Table 3 · p. 15

C Directional Generalization Intervals and Jacobian SVD Analysis

The analyses report that training reshapes directional GIs and local linearity, increases class-specific ray coverage, and produces coverage regions that extend beyond a single train-to-test direction. They also test the robustness and discriminative specificity of these measurements.

C Directional Generalization Intervals and Jacobian SVD Analysis

  • After training, GIs decrease along high-singular-value directions and increase along low-singular-value directions, reaching an average of approximately 11.5.At epoch 1, GIs were relatively uniform at approximately 7; the authors characterize the emerging low-sensitivity directions as an increasingly invariant nuisance space.Source: C Directional Generalization Intervals… · p. 16

D Linear Approximation Analysis Along Singular Vector Directions

  • Training changes local linearity from uniformly short approximation ranges of 0.2–0.4 to ranges near 1.2 along low-singular-value directions.The trained model’s transition around RSV indices 300–400 corresponds to where its linearity becomes markedly better than the untrained model’s.Source: D Linear Approximation Analysis Along… · p. 16

D.1 Remark: Theorem 1 and the Residual Bound

  • The GI lower bound accommodates nonlinearity through a residual bound, and the reported GIs are evaluated on the nonlinear network rather than a linearized substitute.A larger residual reduces the lower bound but does not invalidate the theorem; the representation’s linear approximation range grows to approximately 1.2 after training along low-curvature directions with large GIs.Source: §D.1 · p. 16

E Test-to-Train Generalization Interval Experimental Protocol

  • Test-to-train coverage compares test samples with predicted-class training samples, using all MNIST test cases but sampled test and training sets for CIFAR-10 and ImageNette.The respective protocols use 50 test samples per class and 200 training samples for CIFAR-10, and 20 test samples per class and 100 training samples for ImageNette.Source: E Test-to-Train Generalization… · p. 17

F Robustness and Discriminative Validity of Ray Coverage

  • The robustness checks assess training variability and class specificity by comparing trained and untrained networks and same-class and different-class rays.The different-class condition holds the predicted class fixed and evaluates rays toward three random wrong-class training pools per test sample.Source: F Robustness and Discriminative… · p. 17, §F.1 · p. 17

F.1 Discriminative Baselines: Untrained Networks and Wrong-Class Rays

  • Trained same-class coverage was 89–96%, versus 0.3–1.2% for trained different-class rays; training raised same-class coverage from 32–47%.The results show a larger class-specific effect after training than the mild same-class pixel bias observed in untrained networks.Source: §F.1 · pp. 17–18, Table 4 · p. 17

F.2 Statistical Robustness Across Seeds and Optimizers

  • Final-epoch coverage converged across seeds and optimizers with CV ≤1.8%, while the untrained baselines varied substantially.The untrained-to-final difference was significant in 15 of 16 runs; Table 5 summarizes the seed and optimizer pattern.Source: §F.2 · p. 18, Table 5 · p. 18

F.3 Orthogonal Tube Test: Full Protocol and Results

  • On both tested CNNs, high-coverage rays had an orthogonal stable region at least as wide as the ray itself in every sampled perpendicular direction and location.The tube test sampled 20 perpendicular directions at five points on each ray; a ratio of at least 1 indicates this width relationship.Source: §F.3 · p. 18, Table 6 · p. 18

G Class Region Analysis Along Train-to-Test Paths

Across train-to-test paths, the population-error analysis, interventions, and distributional tests, the paper examines how Generalization Intervals relate to stable predictions and test-time coverage. The reported patterns include an error decrease under a no-re-entry assumption, accuracy changes when coverage is manipulated, and persistent coverage under randomized labels and class imbalance.

G Class Region Analysis Along Train-to-Test Paths

  • GI > 1.0 paths show uninterrupted class predictions, whereas GI < 1.0 paths often show multiple transitions and fragmented, unstable regions.The analysis traces linear paths from MNIST-MLP training samples to same-class test samples.Source: G Class Region Analysis Along… · p. 19

H.1 Derivation of the Population Error Identity

  • When GI increases almost surely, population error decreases or stays unchanged under the no-re-entry assumption; without it, this conclusion can fail.The assumption requires that predictions do not return to the original class after leaving the GI region.Source: §H.1 · p. 20

H.2 Interventional Experiment: Design

  • The intervention expands or restricts local prediction stability by adding a hinge loss on randomly directed perturbations around training inputs.The expand arm encourages confident correct predictions at perturbed inputs; the restrict arm encourages confident misclassification while cross-entropy targets the unperturbed input.Source: §H.2 · pp. 20–21

H.3 Interventional Experiment: Full Results

  • On Fashion-MNIST-CNN, restriction lowers accuracy by 7.4pp and ray coverage by 19.5pp, while expansion raises accuracy by 1.1pp.The changes are reported as statistically significant; on MNIST-MLP, restriction lowers test accuracy (p = 0.003 vs. control).Source: §H.3 · p. 21

I.1 Random Label Experiment Protocol

  • With randomized labels, ray coverage remains the classification mechanism despite averaging 17%–24% lower coverage than with semantically meaningful labels.The experiments include labels reassigned within 10 classes and across 100 classes.Source: §I.1 · p. 21

I.2 Long-tailed Distribution Analysis Details

  • Under long-tailed MNIST, coverage remains 86.6%–94.3% across frequency combinations, while low-frequency samples tend to migrate into higher-frequency classes.In MNIST-LT1, misclassified low-frequency samples migrate to medium-frequency classes (47.2%), high-frequency classes (35.2%), or remain low-frequency (17.6%).Source: §I.2 · pp. 21–22

I.3 Adversarial Examples Experiment Protocol

The adversarial analysis reports model-dependent ray coverage, while the OOD experiment finds nontrivial coverage for CIFAR-10 inputs evaluated by an MNIST-trained MLP. The accompanying proofs relate GI to classification margins, Jacobian sensitivity, and adversarial perturbation size.

I.3 Adversarial Examples Experiment Protocol

  • Adversarial-example coverage varies by model: MNIST-MLP examples show 83.9–86.6% when correctly classified and 91.2–94.5% when misclassified, while ImageNette-ViT-B correct examples show 15.8–19.6%.CIFAR10-ResNet20 coverage is 63.9–68.7% for correctly classified examples and 39.6–58.1% for misclassified examples, depending on attack strength. ImageNette misclassified coverage could not be computed reliably.Source: §I.3 · p. 22

I.4 Out-of-Distribution Sample Analysis

  • CIFAR-10 samples show approximately 10% GI coverage on an MNIST-trained MLP despite being OOD, but their class-index agreement is not semantically meaningful.CIFAR-10 images are converted to grayscale 28 × 28 inputs for the MNIST model.Source: §I.4 · p. 23

J Proof of Theorem 1

  • The proof linearizes the logits at the input and uses the first decision-boundary tie between the true class and a competing class.The expansion is written as f(x0 + ϵd) = f(x0) + ϵJf(x0)d + R.Source: J Proof of Theorem 1 · p. 24

K Proof of Theorem 2

  • Along a right singular vector v_i, the GI lower bound depends on the corresponding singular value σ_i of the penultimate-layer Jacobian.The proof uses Jϕ(x0)v_i = σ_i u_i to bound the output-Jacobian sensitivity.Source: K Proof of Theorem 2 · pp. 24–25

L Proof of Theorem 3

  • The input-space loss gradient is bounded through the chain-rule decomposition of the output Jacobian and the penultimate-layer Jacobian.The proof uses the cross-entropy gradient with respect to logits and bounds its norm before bounding the input gradient.Source: L Proof of Theorem 3 · p. 25

M Proof of Theorem 4

  • Any successful adversarial perturbation has norm at least the minimum GI over directions, with a further lower bound involving the largest singular value σ1.The proof decomposes the perturbation into direction and magnitude before optimizing the bound over directions.Source: M Proof of Theorem 4 · pp. 25–26

N Understanding GI Growth During Training

GI growth during training is anisotropic: it is strongest along high-index right singular vectors with small singular values. The proposed account links this pattern to margin growth and changes in Jacobian sensitivity, while noting that individual singular-value decay is not rigorously established.

N Understanding GI Growth During Training

  • During training, GIs grow most along high-index right singular vectors with small σ_i and shrink along low-index directions with large σ_i.The account identifies increasing margin and decreasing sensitivity as factors that increase the GI lower bound.Source: N Understanding GI Growth During… · p. 26
  • The proposed mechanism combines non-decreasing normalized margin with faster GI growth where σ_i is already small, but rigorous proof of individual σ_i decay remains open.The account attributes the directional pattern to empirical observations of the penultimate-layer Jacobian and prior aggregate sensitivity results.Source: N Understanding GI Growth During… · p. 26
  • Training sharpens Jacobian anisotropy: high-index singular values shrink further, while low-index singular values may grow.The passage describes this as a nonuniform change rather than a uniform shrinkage of all singular values.Source: N Understanding GI Growth During… · p. 26
Loading 2610.23716v1…