PaperlayerView exact revision on arXiv

Source-linked AI summary

The Local Learning Coefficient: A Singularity-Aware Complexity Measure

Edmund Lau, Zach Furman, George Wang, Daniel Murfet, Susan Wei

arXiv:2308.12108v2stat.MLcs.AIcs.LG

TL;DR

Deep neural networks challenge parsimony, while parameter count is not an adequate measure of their complexity. The paper introduces the LLC and a scalable estimator; it reports that common training heuristics control LLC and that stronger implicit regularization corresponds to lower LLC and higher test accuracy. It also notes that ResNet LLC estimates cannot be calibrated against true values.

  • Problem

    Deep neural networks challenge the principle of parsimony because their complex architectures make their effective simplicity difficult to characterize.

    Source: §1 · p. 1

  • Method

    The paper introduces the LLC from local free energy and develops a scalable estimator whose perturbation-based expectation measures loss near a local minimum.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    The paper reports that common training heuristics control LLC, with stronger implicit regularization corresponding to lower LLC and higher test accuracy in ResNet18 on CIFAR10.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The LLC can discern the implicit regularization pressure applied by training heuristics.

    Source: §1 · p. 2

    ResNet18 LLC estimates cannot be calibrated against true LLC values, so the paper uses relative comparisons between models with shared context.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC's theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning's complexity and the principle of parsimony.

1 Introduction

The paper asks how deep neural networks can be reconciled with parsimony when parameter count does not capture their effective complexity. It introduces the Local Learning Coefficient (LLC), an SLT-based measure, and reports that stronger implicit regularization is associated with lower LLC and higher test accuracy.

1 Introduction

  • Parameter count is an inadequate measure of deep-network complexity because information captured by a trained network can be decoupled from its number of parameters.The passage points to results on generalization, pruning, and distillation as evidence.Source: §1 · p. 1
  • The paper motivates a singularity-aware measure because deep neural networks have many parameter variations that leave the represented function unchanged.The paper describes such models as highly singular and degenerate.Source: §1 · pp. 1–2
  • The paper introduces the LLC, develops a scalable estimator, and reports empirical validation of its accuracy and scalability to modern network sizes.The introduction also reports that common training heuristics effectively control LLC.Source: §1 · p. 2
  • Stronger implicit regularization—higher learning rate, lower batch size, or higher momentum—corresponds to lower LLC and higher test accuracy in ResNet18 on CIFAR10.The training loss reaches zero in most instances, so it does not distinguish the effects of implicit regularization.Source: Figure 1 · p. 2, §1 · p. 3

2 Setup

The setup defines the data-generating process, the posited model and prior, and the losses used to study complexity. It treats neural networks within a singular-learning framework and assumes the stated SLT conditions.

2 Setup

  • The setup specifies a true data-generating mechanism q, a posited model p parameterized by network weights, and a prior φ over weights.The training data consist of n input-output pairs drawn i.i.d. from q.Source: §2 · p. 3
  • It defines sample negative log likelihood as training loss and expected negative log likelihood under q as population loss.These losses encompass common choices such as mean squared error and cross-entropy.Source: §2 · p. 3
  • A model is regular when its parameter-to-distribution map is one-to-one and its Fisher information is everywhere positive definite; otherwise it is singular.The setup assumes the model-truth-prior triplet satisfies fundamental SLT conditions, discussed in Appendix A.1.Source: §2 · p. 3

3 The local learning coefficient

The LLC extends the global learning coefficient to describe complexity near a local loss minimum. It uses low-loss volume scaling to account for degenerate loss geometry, and the paper states that it is invariant under local smooth, invertible reparameterizations.

3 The local learning coefficient

  • The LLC extends the global learning coefficient and is introduced as an invariant based on the volume of the loss-landscape basin near a parameter.The paper uses this geometric perspective to explain the measure.Source: §3 · p. 3

3.1 Complexity via counting low loss parameters

  • Near a local minimum, complexity can be framed as the number of bits needed to specify the minimum within a loss tolerance.The low-loss parameter set is defined within a ball around the minimum, where loss is no lower than at the minimum.Source: §3.1 · p. 3
  • The LLC is the asymptotic volume-scaling exponent near a minimum: multiplying the error tolerance by a scales volume by a^λ(w∗).The figure contrasts λ=1 for a quadratic loss with λ=1/4 for a degenerate loss, whose low-loss region has more volume.Source: §3.1 · p. 4, Figure 2 · p. 4
  • Unlike the regular case, the scaling exponent λ(w∗) depends on the underlying data distribution q(x, y).Source: §3.1 · p. 5
  • The LLC is invariant to local diffeomorphisms, so locally smooth, invertible changes of variables do not alter its value.This is motivated by the aim to assign the same complexity to models that differ only in representation.Source: §3.1 · p. 5

4 LLC estimation

The LLC estimator uses the asymptotic local free-energy expansion to estimate complexity near a parameter-space minimum. The paper develops an idealized estimator, practical approximations, and a minibatch SGLD implementation.

4 LLC estimation

  • The estimation strategy leverages the LLC’s appearance in the asymptotic expansion of local free energy, connecting volume scaling to a practical estimate.The paper presents an idealized estimator as theoretically sound but computationally intractable, then develops an implementable version.Source: §4 · p. 5

4.1 Idealized LLC estimator

  • For singular models, local free energy asymptotically contains λ(w*); rearranging this expansion motivates an idealized LLC estimator.Estimating its local posterior free energy directly is difficult because sampling must be confined to a neighborhood of w*.Source: §4.1 · p. 5

4.2 Surrogate for enforcing Bγ(w∗)

  • A localizing Gaussian prior centered at w* serves as a proxy for restricting integration to Bγ(w*), with large γ keeping the posterior close to w*.Its normalizing constant is used as a proxy for the neighborhood’s free energy.Source: §4.2 · p. 6

4.3 The LLC estimator

  • Setting β*=1/log n makes the localized posterior expectation’s leading asymptotic terms match the surrogate free energy used in the LLC estimator.The estimator combines the neighborhood-free-energy approximation with this expectation; its value depends on γ.Source: §4.3 · p. 6
  • The plug-in estimator uses the training dataset twice, but the authors report no adverse effects in their experiments.This occurs when the empirical-loss minimizer is substituted into the estimator.Source: §4.3 · p. 6

4.4 The SGLD-based LLC estimator

  • Minibatch SGLD makes posterior sampling less costly than traditional full-data MCMC, whose gradients require the entire training dataset.The stated cost is roughly one SGD step per required sample.Source: §4.4 · p. 7
  • After burn-in, approximate tempered-posterior samples from SGLD support the estimator, while minibatch forward passes provide unbiased estimates of full-data loss.The forward passes used to compute minibatch loss for gradients are recycled for this estimate.Source: §4.4 · p. 7

5 Experiments

The experiments test LLC estimator accuracy and scalability, then examine how regularization settings relate to LLC in ResNet18. Deep linear networks provide a setting with theoretical learning coefficients for comparison.

5 Experiments

  • The experiments use a minibatch SGLD-based LLC estimator, with additional appendix studies covering transformers, sampler gradients, optimizers, and scaling invariance.The listed appendix experiments include transformer language models, minibatch-versus-full-batch gradients, SGD versus entropy-SGD, and a scaling-invariance check.Source: §5 · p. 7

5.1 LLC for Deep Linear Networks (DLNs)

  • Deep linear networks offer a testbed because theoretical LLC values are available, and their estimator can be checked against them at modern network sizes.The networks are also used as a sandbox for deep learning theory; Aoyagi (2024) clarified theoretical LLCs for DLNs.Source: §5.1 · p. 8
  • 100M parameters: the LLC estimator accurately estimates learning coefficients in deep linear networks, including when evaluated at an SGD-found local minimum rather than a population-loss minimum.Figure 3 compares estimates with true learning coefficients; accuracy is maintained in the non-global-minimum setting.Source: Figure 3 · p. 8, §5.1 · p. 8

5.2 LLC for ResNet

  • On ResNet18 trained on CIFAR10, higher learning rate, lower batch size, and higher momentum accompany higher test accuracy and lower LLC, while training loss usually reaches zero.The factors are varied in isolation; stronger explicit L2 regularization is also accompanied by lower LLC.Source: §5.2 · pp. 8–9
  • ResNet18 LLC estimates cannot be calibrated against true values, so the experiments use relative comparisons between models with shared context.The paper suggests holding other factors constant while varying one setting, such as batch size, learning rate, or momentum.Source: §5.2 · p. 9

6 Related work

The paper situates LLC within Singular Learning Theory and distinguishes it from both broad function-class complexity measures and related finite-sample or flatness measures.

6 Related work

  • The work’s theoretical foundation is Singular Learning Theory (SLT); its global learning coefficient, introduced by Watanabe (2001), provides the asymptotic expansion of free energy.Free energy is equivalent to the negative log Bayes marginal likelihood; Watanabe (2013) introduced WBIC as a free-energy estimator.Source: §6 · p. 9
  • The LLC is a singularity-aware basin-broadness measure related to PAC-Bayes flatness measures [Neyshabur et al., 2017], accounting for non-Gaussian posteriors in singular models.The paper contrasts this with ordinary flatness or sharpness approaches that do not account for that posterior structure.Source: §6 · p. 9
  • The LLC measures complexity for a specific model p(y|x, w), unlike Rademacher complexity and VC dimension, which concern a function class.This model-specific focus is presented as useful for studying interactions among function class, data properties, and training heuristics.Source: §6 · p. 9
  • Concurrent work proposes learning capacity as a finite-n version of the learning coefficient and studies its dependence on training size n [Chen et al., 2023a].Source: §6 · p. 9

7 Outlook

The outlook frames LLC within a broader SLT account of singular models and develops conditions and properties that support its local interpretation. It points to possible applications in studying learning dynamics and phase transitions.

7 Outlook

  • The paper proposes that a free-energy perspective may help unify accounts of deep-network learning dynamics by balancing model fit and complexity.It identifies phase transitions, emergent abilities, and saddle-to-saddle learning as directions for future study.Source: §7 · pp. 9–10

A Background on Singular Learning Theory

  • SLT treats the learning coefficient as a model-class complexity quantity, while the LLC is its local counterpart at a particular parameter minimum.Singular models have degenerate parameter settings whose neighborhoods influence learning asymptotics; the RLCT is the learning coefficient under the stated condition φ > 0 on W0.Source: A Background on Singular Learning Theory · pp. 14–15

A.1 Background assumptions in SLT

  • The stated SLT assumptions are sufficient rather than necessary, and the authors note that relative finite variance may be unclear for experiments on real data.When the true density is realizable by the model, the relative finite variance conditions are automatically satisfied.Source: §A.1 · p. 15

B Well-definedness of the theoretical LLC

  • Under the local assumptions, the local free energy has λ(w*) as its log n coefficient, linking the local learning coefficient to its asymptotic expansion.The derivation assumes w* is at least as degenerate as any nearby minimizer with the same loss.Source: B Well-definedness of the theoretical… · p. 16

C Reparameterization invariance of the LLC

D Consistency of local WBIC

  • Local WBIC matches the first two terms of the local free-energy expansion, but this does not immediately establish that it estimates free energy over a neighborhood.The authors conjecture that its estimated coefficient equals the LLC under conditions on γ, leaving those conditions for future theoretical work.Source: D Consistency of local WBIC · p. 17

E Related work

The section positions LLC as a trained-network, data-sensitive complexity measure that accounts for singular degeneracy and differs from task-intrinsic complexity, and presents estimator pseudocode, sampling guidance, and deep-linear-network results. It also describes how increasing depth can make these networks more singular even as parameter count grows.

E Related work

  • The LLC measures a trained function rather than an entire function class, depends on the data distribution, and handles general DNN degeneracy beyond Laplace-based flatness accounts.The estimator is framed as a localized WBIC extension; LLC also quantifies local identifiability, while singular posteriors challenge Gaussian PAC-Bayes flatness choices.Source: E Related work · pp. 17–19

F Model complexity vs model-independent complexity

  • LLC is model complexity, not model-independent task complexity; different models can assign different LLCs, though neural-network universality is a possible future direction.The paper cautions that LLC alone does not explain neural networks’ simplicity bias.Source: F Model complexity vs… · pp. 19–20

G SGLD-based LLC estimator: minibatch version pseudocode

  • The minibatch SGLD estimator initializes at w*, samples with temperature β*=1/log n, and uses mean log likelihood to estimate WBIC and the loss at w*.Its pseudocode draws random minibatches and adds Langevin noise during iterative parameter updates.Source: G SGLD-based LLC estimator: minibatch… · p. 20

H Recommendations for accurate LLC estimation and troubleshooting

  • The practical estimation recommendations are largely based on experience with deep linear networks, the only realistic wide-and-deep model here with a known theoretical learning coefficient.Source: H Recommendations for accurate LLC… · p. 21

H.1 Step size

  • Tune the SGLD step size using MALA acceptance values as a diagnostic, targeting an average of 0.9–0.95 rather than applying an acceptance correction.Values below the range suggest increasing the step size; values above it suggest decreasing it for efficiency.Source: §H.1 · p. 21

H.2 Step count and burn-in

  • Run sampling until the loss trace is truly flat, discard burn-in samples, and tally loss only after flattening to avoid underestimating LLC.An insufficient chain can underestimate LLC; a slow upward loss trend may still signal underestimation.Source: §H.2 · pp. 21–22

H.3 Other issues and troubleshooting

  • Negative LLC estimates can arise when trajectories reach lower-loss regions, while an overly large restoring force can overwhelm the likelihood gradient and erase sensitivity to local geometry.For pathological cases, the authors recommend keeping γ small and gradually lowering the step size while lengthening the chain enough for the loss trace to equilibrate.Source: §H.3 · p. 22

I Learning coefficient of DLN (Aoyagi, 2024)

  • The theoretical learning coefficient for deep linear networks decreases with network depth, and it is conceptually distinct from the LLC measured at particular minima.The global learning coefficient is the minimum LLC among global minima of the population loss.Source: I Learning coefficient of DLN (Aoyagi,… · p. 24
  • Aoyagi (2024) provides further theoretical examples of the phenomenon.The passage points to Figure 1 in Aoyagi (2024).Source: I Learning coefficient of DLN (Aoyagi,… · p. 24
  • As DLN depth increases, λ decreases despite rising parameter count, and most randomly generated networks are singular, with multiplicity m greater than 1.The plot uses networks with 2–800 hidden layers and random widths of 100–2,000, including the input dimension; each network’s rank is drawn from zero to the minimum layer width.Source: I Learning coefficient of DLN (Aoyagi,… · p. 24, Figure I.1 · p. 25

J Experiment: DLN

The DLN experiments test LLC estimates on randomly generated architectures and synthetic data across model-size scales, while supplementary plots and ResNet experiments examine estimation behavior and training configurations.

J Experiment: DLN

  • The DLN study randomly varies layer counts, widths, and sometimes weight-matrix ranks, then generates synthetic training data from each network’s true parameter.Configuration ranges and SGLD settings vary by target model-size order of magnitude; larger models tend to require smaller step sizes and more steps.Source: J Experiment: DLN · p. 26

J.1 Hyperparameters and further details

  • DLN experiments use tuned SGLD settings and synthetic data, but some chains blow up and mean MALA acceptance varies across runs.The text notes that model size alone does not determine a suitable step size, and suggests µ-parameterization as a possible future way to alleviate tuning with scale.Source: §J.1 · p. 26, J Experiment: DLN · p. 26, Figure J.3 · p. 28

J.2 Additional plots for DLN experiments

  • Figure J.1 presents estimated LLC against true learning coefficients in linear scale for experiments spanning different DLN model-size ranges.Each plot represents one experiment batch, with SGLD hyperparameters tuned once per batch.Source: §J.2 · p. 26, Figure J.1 · p. 27
  • Figure J.2 plots relative error between estimated LLC and theoretical learning coefficients across DLN model-size orders of magnitude.The relative error is defined as (λ − ˆλ(w∗))/λ.Source: §J.2 · p. 26, Figure J.2 · p. 28

K Experiment: LLC for ResNet

  • The ResNet experiments vary learning rate, batch size, momentum, and L2 regularization while tracking LLC at training checkpoints.Each optimizer-configuration group uses a calibration run to select shared SGLD hyperparameters; rare negative estimates or divergent chains are removed and widen confidence intervals.Source: K Experiment: LLC for ResNet · p. 29

K.1 Details for main text Figures 1

  • Figure 1 separately varies learning rate, batch size, and SGD momentum under stated fixed-setting baselines; LLC estimates are taken every 1000 iterations for learning-rate/momentum sweeps and 2500 for batch-size sweeps.Learning-rate values were [0.005, 0.05, 0.01, 0.1, 0.2] (batch size 512, no momentum, 30000 iterations); batch sizes were [16, 32, 64, 128, 256, 512, 1024] (learning rate 0.01, no momentum, 100000 iterations); momentum values were [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] (learning rate 0.05, batch size 512, 20000 iterations). Each experiment was repeated with 5 different random seeds.Source: §K.1 · p. 29, K Experiment: LLC for ResNet · p. 29

K.2 Additional ResNet18 + CIFAR10 LLC experiments

These additional ResNet18-on-CIFAR10 experiments vary optimizer momentum and explicit L2 regularization, tracking LLC estimates during training.

Experiments using SGD with momentum

  • The experiments repeat the learning-rate and batch-size variations from Figure 1 (top and middle), training ResNet18 on CIFAR10 with SGD momentum; Figure K.1 presents the corresponding results.Figure K.1 contrasts these runs with the no-momentum experiments in Figure 1.Source: §K.2 · p. 29, Figure K.1 · p. 31

Explicit L2-regularization

  • Explicit L2 regularization is added to the cross-entropy training loss, with regularization rate α varied for SGD runs with and without momentum.LLC is estimated using the original cross-entropy loss without the added L2 term.Source: §K.2 · p. 30
  • The L2-regularization experiments track LLC at evenly spaced checkpoints for both momentum settings.Figure K.2 is described as showing the impact of varying the explicit L2-regularization rate.Source: §K.2 · p. 30, Figure K.2 · p. 31

L Additional Experiment: LLC for language model

The paper estimates LLC for an attention-only transformer trained on a subset of the Pile. The reported estimates are stable across runs and small relative to the model’s parameter count.

L Additional Experiment: LLC for language model

  • The transformer LLC estimates are reported as remarkably stable across multiple training runs.The experiment’s stated purpose is to verify consistent LLC estimation for a transformer trained on language data.Source: L Additional Experiment: LLC for… · p. 32
  • The end-of-training LLC estimates are a tiny fraction of the transformer’s 3.3m parameters.The estimates are reported over 10 training repetitions for an attention-only transformer trained on a subset of the Pile.Source: L Additional Experiment: LLC for… · p. 32
  • The tested model is a two-layer attention-only transformer with 3,355,016 parameters, trained on a resampled Pile subset.It uses a truncated 5,000-token GPT-2 vocabulary and a context length of 1024.Source: L Additional Experiment: LLC for… · p. 32

M Additional Experiment: MALA versus SGLD

The paper compares minibatch-SGLD LLC estimates with MALA-based estimates on synthetic realizable data across dataset sizes. The estimates agree, while SGLD has lower computational cost, especially as dataset size grows.

M Additional Experiment: MALA versus SGLD

  • The comparison tests whether minibatch loss in SGLD sampling and LLC calculation affects estimation, against the MALA-based estimate.The authors describe the setup as stacking the odds against the minibatch-SGLD estimator.Source: M Additional Experiment: MALA versus… · p. 33
  • Experiments use a realizable two-hidden-layer ReLU network and sweep dataset sizes from 100 to 100000.Inputs are uniformly generated and labels are noiseless outputs from the true network; both estimators share the same randomly chosen true parameter.Source: M Additional Experiment: MALA versus… · p. 33
  • SGLD and MALA LLC estimates agree across dataset sizes, while the SGLD-based estimate has far lower computational cost, especially as datasets grow.The comparison summarizes the reported results in Figure M.1.Source: Figure M.1 · p. 33

N Additional Experiment: SGD versus eSGD

Entropy-SGD finds minima with lower LLC than SGD, while both optimizers yield estimates far below the network’s parameter count. The section also reports that the LLC estimator appears invariant to ReLU rescaling when preconditioning is used.

N Additional Experiment: SGD versus eSGD

O Additional Experiment: scaling invariance

  • The LLC is intended to ignore function-preserving ReLU rescaling symmetries, and the estimator appears to do so empirically when properly preconditioned.The empirical verification covers a specific easy-to-test case, not the property in general.Source: O Additional Experiment: scaling… · p. 35

O.1 Preconditioned SGLD

  • Rescaling makes the loss landscape sufficiently anisotropic that the original SGLD sampler requires prohibitively small steps; practical preconditioning must be learned adaptively.The experiment instead uses a fixed preconditioning matrix for convenience.Source: §O.1 · p. 35

O.2 Experiment

Across eight orders of magnitude of ReLU rescaling, LLC estimates appear invariant, with small error bars across SGLD runs. The compute disclosure reports GPU-hour totals for four experiment groups.

O.2 Experiment

  • Across eight orders of magnitude of rescaling, LLC estimates appear invariant, with small error bars across SGLD runs.The experiment uses preconditioning and full-batch loss to calculate LLC.Source: §O.2 · p. 35, Figure O.1 · p. 36

P Compute resources disclosure

  • Estimated totals were 287 GPU hours across 208 experiments; 150 GPU hours across 600 experiments, including failed ones; 27 GPU hours for two sets of 80 repetitions; and 5 GPU hours for 10 repetitions.The 600 experiments were each around 15 GPU minutes; the 27-hour estimate uses 10 GPU minutes per repetition, and the 5-hour estimate uses 30 GPU minutes per repetition.Source: P Compute resources disclosure · p. 37
Loading 2308.12108v2…