PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition

Zhongtian Chen, Edmund Lau, Jake Mendel, Susan Wei, Daniel Murfet

arXiv:2310.06301v1cs.LGcs.AI

TL;DR

The paper asks how observed plateaus and sudden loss drops during SGD training in TMS relate to mathematically defined Bayesian phase transitions. Using Singular Learning Theory and experiments, it finds k-gon-dominated Bayesian transitions and SGD movement toward lower-loss, higher-complexity solutions, while noting that Bayesian and dynamical transitions need not correspond one-to-one.

  • Problem

    In TMS, SGD training shows plateaus and sudden loss drops, but the formal status of these dynamical transitions and their relation to Bayesian posterior phase transitions remain unresolved.

    Source: §1 · p. 1

  • Method

    The paper derives a high-sparsity population loss and uses Singular Learning Theory, k-gon critical points, local learning coefficients, posterior sampling, and SGD experiments to examine both kinds of transitions.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    For two hidden dimensions, Bayesian transitions occur between k-gon-dominated phases, while SGD empirically moves from high-loss, low-complexity solutions toward low-loss, high-complexity ones.

    Source: §1 · pp. 1, 3

  • Takeaways & Limitations

    In the studied TMS setting, the local learning coefficient helps characterize Bayesian phase preferences and reveals a sequential learning pattern in SGD consistent with a proposed relation between the transitions.

    Source: §1 · p. 3, §6 · p. 9

    A Bayesian transition need not have an associated SGD transition; the 5→6 Bayesian transition, for example, has not been observed dynamically.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

We investigate phase transitions in a Toy Model of Superposition (TMS) using Singular Learning Theory (SLT). We derive a closed formula for the theoretical loss and, in the case of two hidden dimensions, discover that regular $k$-gons are critical points. We present supporting theory indicating that the local learning coefficient (a geometric invariant) of these $k$-gons determines phase transitions in the Bayesian posterior as a function of training sample size. We then show empirically that the same $k$-gon critical points also determine the behavior of SGD training. The picture that emerges adds evidence to the conjecture that the SGD learning trajectory is subject to a sequential learning mechanism. Specifically, we find that the learning process in TMS, be it through SGD or Bayesian learning, can be characterized by a journey through parameter space from regions of high loss and low complexity to regions of low loss and high complexity.

1 Introduction

The paper connects Bayesian phase transitions in TMS, which are mathematically defined in Singular Learning Theory, to the observed plateaus and sudden loss drops during SGD training. It derives theoretical tools for identifying critical points and reports evidence that both Bayesian learning and SGD move from high-loss, low-complexity regions toward low-loss, high-complexity ones.

1 Introduction

  • For two hidden dimensions, Bayesian phase transitions occur between phases dominated by regular-polygon weight configurations, or k-gons, whose local learning coefficients and losses determine posterior preferences.The paper derives theoretical local learning coefficients for commonly encountered k-gons and verifies the predicted phases and transitions by posterior sampling.Source: §1 · pp. 1–2
  • SGD training exhibits plateaus separated by sudden loss drops, with each drop accompanied by a jump in estimated local complexity as training tolerates more complex solutions.This pattern motivates the Bayesian antecedent hypothesis, proposing that Bayesian phase transitions stand behind dynamical transitions.Source: §1 · p. 2
  • The paper derives a closed-form population loss and partially classifies its critical points, then uses their local learning coefficients to study phase behavior.It also experimentally tests the predicted Bayesian transition and examines dynamical transitions during SGD.Source: §1 · pp. 2–3
  • In TMS, both Bayesian learning and SGD are characterized as moving from high-loss, low-complexity regions toward low-loss, high-complexity regions.Complexity in the SGD experiments is measured by the estimated local learning coefficient.Source: §1 · pp. 1, 3

2 Related work

Prior work describes optimization in deep linear and multi-index models as moving through solutions of decreasing loss and increasing complexity. This paper situates TMS in that line of inquiry while distinguishing SLT's theoretically justified account of entropy-energy competition from an earlier approach criticized for misapplying the Laplace approximation.

2 Related work

  • In deep linear networks, optimization trajectories tend toward lower-loss, higher-complexity solutions, with gradient-based optimization inducing low-rank solutions in deep matrix factorization.The related work describes saddle-to-saddle dynamics through neighborhoods of saddles of decreasing index.Source: §2 · p. 3
  • For a restricted multi-index model, SGD adaptively selects target functions of increasing complexity, with low-complexity solutions preferred early in training.The cited result depends on restrictions on the input data distribution.Source: §2 · p. 3
  • A proposed entropy-energy account of these optimization intuitions is criticized for lacking theoretical justification because it applies the Laplace approximation incorrectly.The cited works are Zhang et al. (2018) and Wei et al. (2022b); Lau et al. (2023).Source: §2 · p. 3
  • The related work presents SLT's internal model selection principle as a theoretically grounded form of entropy-energy competition for neural networks.It is offered as potential theoretical backing for intuitions developed in the deep-linear-network literature.Source: §2 · p. 3

3 Toy Model of Superposition

The paper derives the TMS population loss in the high-sparsity limit, identifies k-gon critical points for two hidden dimensions, and characterizes their losses and local learning coefficients. These results provide the geometric quantities used to analyze Bayesian phase transitions.

3.1 The TMS Potential

  • In the high-sparsity limit, the TMS setup yields a closed-form population loss, called the TMS potential.The potential is analytic at many relevant critical points for r = 2, but not at the 4-gons.Source: §3.1 · pp. 3–4

3.2 k-gon critical points

  • For r = 2, the paper proves that various k-gons are critical points of the TMS potential, using rotational and joint column-bias permutation symmetries.The classification includes standard k-gons and variants; for c in {5, 6, 7, 8}, the standard c-gon is non-degenerate, while for k in {5, 6, 7, 8} with k < c it is minimally singular.Source: §3.2 · pp. 4–5

3.3 Local learning coefficients

  • The theoretical local learning coefficients and losses of selected critical points are summarized for the r = 2, c = 6 TMS potential.For minimally singular points including 5-, 5+-, and 6-gons, the coefficient agrees with half the number of normal directions to the local level-set manifold; moving from 5-gon to 6-gon fixes one column and its bias, reducing three free parameters.Source: Table 1 · p. 5, §3.3 · p. 5

4 Bayesian Phase Transitions

Singular Learning Theory predicts that posterior preferences across parameter-space regions can shift with sample size as loss and local learning coefficient jointly determine free energy. For r = 2, c = 6, theory and MCMC support a 5-to-6-gon transition around n = 600–700, though the asymptotic theory's finite-sample use and low-sample classifications have limits.

4 Bayesian Phase Transitions

  • In singular models, Bayesian learning can involve posterior jumps between parameter-space regions at critical sample sizes, unlike concentration at a MAP estimator in regular models.The paper focuses on transitions as sample size varies, not transitions induced by changing the prior or true distribution.Source: §4 · p. 5

4.1 Internal Model Selection

  • The free-energy formula combines fit and complexity to describe local posterior preference, and a phase transition occurs when the ordering of two regions swaps as sample size changes.Generically, increasing sample size favors phases with lower loss and higher complexity.Source: §4.1 · pp. 6–7

4.2 Experiments

  • The experiments define regions by the number of convex-hull points in the weight columns and the number of positive biases, then test whether the corresponding k-gon dominates each region's posterior.The paper reports that the chosen regions are appropriate for the sample-size range considered, while high-loss exotic k-gons that may dominate at very low n are outside the experiments' scope.Source: §4.2 · p. 7
  • The 5→6 transition is predicted near n = 600 and observed in MCMC between n = 600 and 700 for r = 2, c = 6.The free-energy analysis is asymptotic in n; below n = 400, MCMC classification becomes increasingly uncertain, so correspondence with theory should not be overinterpreted.Source: §4.2 · p. 7, Figure 2 · p. 7

5 Dynamical Phase Transitions

For r = 2, c = 6, SGD trajectories show plateaus associated with classified critical points, while transitions move toward lower loss and higher estimated complexity. Bayesian and dynamical transitions are related but do not necessarily correspond one-to-one.

5 Dynamical Phase Transitions

  • The trajectories are characterized by plateaus associated with the critical points described earlier, and the dominance of these points is also visible in Figure 3.The reported relationship is decreasing loss alongside increasing complexity.Source: §5 · p. 8

5.1 Relation Between Bayesian and Dynamical Transitions

  • The same k-gon critical points dominate Bayesian posterior phases and explain plateaus in SGD training.The paper relates this to SLT's association of posterior phases with KL-divergence singularities and nonlinear dynamics around potential singularities.Source: §5.1 · p. 9
  • A Bayesian transition need not produce an SGD transition; the 5 → 6 Bayesian transition was not observed dynamically and may be rare per SGD step.Distant regions or high energy barriers are offered as possible reasons for the missing dynamical transition.Source: §5.1 · p. 9
  • The authors propose the Bayesian Antecedent Hypothesis: dynamical transitions encountered in neural network training have Bayesian antecedents.Source: §5.1 · p. 9
  • The analysis suggests Bayesian antecedents for the observed transitions, but is inconclusive for 4++−−−→4+−− and 4+−−−→4−−.Although transitions in (10) increase estimated local learning coefficient, low-sample-size free-energy constants may matter.Source: §5.1 · p. 9

6 Conclusion

The paper reports phase transitions in both Bayesian and stochastic-gradient learning for the two-hidden-dimension TMS, with k-gon critical points dominating both. Its appendix describes 18 critical-point families for r = 2, c = 6 and explains variations that preserve their potential values.

6 Conclusion

  • In the high-sparsity limit, TMS with two hidden dimensions exhibits phase transitions in both stochastic-gradient and Bayesian learning, with phases dominated by classified k-gon critical points.Source: §6 · p. 9
  • The authors propose relating SGD transitions to Bayesian posterior phase transitions through the Bayesian Antecedent Hypothesis.Source: §6 · p. 9
  • The local complexity measure λ̂ reveals a sequential learning mechanism in TMS SGD, consistent with observations in deep linear networks and multi-index models.The authors say λ̂ was not engineered specifically for TMS and could in principle be used to study training in other neural networks.Source: §6 · p. 9

A Fantastic Critical Points and Where to Find Them

  • Known critical parameters are classified by polygon vertex count k, positive-bias count σ, and large-negative-bias count φ; φ > 0 has so far appeared only for 4-gons.For r = 2, c = 6, these descriptions and constraints yield 18 families of critical points.Source: A Fantastic Critical Points and Where… · p. 11
  • Irregular parameter variations can remain on the same critical submanifold and have the same potential value as their regular counterparts.The described examples mostly arise because negative biases allow extra variability without changing the loss value.Source: A Fantastic Critical Points and Where… · p. 11
  • For standard 5-gons, a vestigial bias may be any negative value if its corresponding weight-column length stays below the stated bound.The bound is min{|b′|, |b*|}, where b* is the optimal negative bias for the main columns.Source: A Fantastic Critical Points and Where… · p. 11

B Additional examples of SGD trajectories for the r = 2, c = 6 TMS potential

The additional r = 2, c = 6 SGD trajectories with transitions pass through classified critical points, and the estimated local learning coefficient jumps at each transition. The appendix also describes how to read the trajectory figures and notes an estimator caveat for degenerate k-gons.

B Additional examples of SGD trajectories for the r = 2, c = 6 TMS potential

  • In all reported runs with transitions among 30 random seeds, plateau critical points fit the classification, and the estimated local learning coefficient jumps at each transition.Source: B Additional examples of SGD… · p. 15
  • The trajectory figures pair weight-vector visualizations and quantitative column views with training-time markers, losses, and estimated local learning coefficient.The estimate is smoothed over six observations and measured every 30 epochs.Source: B Additional examples of SGD… · p. 15
  • For particularly degenerate k-gons, the fixed estimator setting ε = 0.001 can yield negative λ̂ values; the figures nevertheless use fixed SGLD hyperparameters.The authors say adapting ε to degeneracy can correct the estimates, but they cannot predict SGD iterates in advance.Source: B Additional examples of SGD… · p. 15

C Using the Free Energy Formula

The appendix uses an asymptotic free-energy approximation to identify Bayesian phase transitions by comparing phase losses, local learning coefficients, and constant terms. Its predictions depend on the approximation’s range of validity and, in some cases, on constants that can materially shift transition sizes.

C Using the Free Energy Formula

  • A transition is predicted when the free-energy difference equation n∆L + ∆λ log n + ∆c = 0 has a positive solution.The approximation is intended to describe the posterior only for sufficiently large n, so whether a predicted transition corresponds to an actual posterior transition remains empirical.Source: C Using the Free Energy Formula · p. 22

C.1 Constant terms in the Free Energy Formula

  • The prior contributes to a phase’s constant term through its density and the phase volume, alongside an O(1) term.The analysis proceeds assuming the prior supplies the leading contribution, although the constant and lower-order terms are not well understood; some displayed prior contributions are lower bounds because negative biases can produce arbitrarily large negative biases.Source: §C.1 · p. 22, C Using the Free Energy Formula · p. 22

C.2 Theoretical predictions for the 5-gon to 6-gon transition for r = 2, c = 6

  • For the 5-gon to 6-gon transition, ∆c = 2.7535 gives a predicted critical sample size of n_cr = 601.This value is obtained by solving the free-energy difference equation numerically.Source: §C.2 · p. 22

C.3 Influence of constant terms

  • For the analyzed transitions, |∆λ/∆L| primarily determines n_cr, while constant terms can still significantly influence it.Here ∆L is about 0.01 and ∆λ and ∆c are about 1; in the range 200 ≤ n ≤ 1000, the loss term is roughly an order of magnitude smaller than the learning-coefficient term.Source: §C.3 · p. 23
  • For the 4+ to 5 transition, including ∆c shifts predicted n_cr from about 327 to about 380.The constant term therefore shifts the critical sample size by a lower-order contribution in this example.Source: §C.3 · p. 23

C.4 Double Transitions

  • Along a sequence of Bayesian transitions, the slope in the (λ, L) plane must increase.For the analyzed sequence, successive slopes become less negative; the change is least clear for the closely spaced 4+−− to 4+ and 4+ to 5 transitions.Source: §C.4 · pp. 23–24

D Bayesian Antecedents

  • When ∆c = 0 and loss decreases, a positive critical sample size exists if and only if the local learning coefficient increases.The critical sample size is then uniquely determined.Source: D Bayesian Antecedents · p. 24
  • When ∆c is nonzero, the analysis checks whether free-energy curves intersect; mixed-sign cases are harder to assess generally.The authors expect a solution when both ∆c and ∆λ are positive, and none when both are negative, given the stated orders of magnitude.Source: D Bayesian Antecedents · p. 24

D.1 The BAH for r = 2, c = 6

For r = 2, c = 6, the observed dynamical transitions move toward lower loss and higher estimated local learning coefficient. Free-energy comparisons suggest Bayesian antecedents, but low-sample-size predictions are sensitive to uncertain terms.

D.1 The BAH for r = 2, c = 6

  • Observed dynamical transitions move down and to the right in the (λ, L) plane, indicating positive critical sample sizes if constant terms are ignored.The loss values come from the critical-point analysis, while some learning coefficients are estimates with stated caveats.Source: §D.1 · p. 24, Figure D.1 · p. 25
  • At low n, predicted Bayesian transitions are sensitive to small changes in learning coefficients and constant terms, so their existence is uncertain.The plotted transitions at low n are described as merely suggestive or weak evidence.Source: §D.1 · p. 24, Figure D.2 · p. 25, Figure D.3 · p. 26
  • Free-energy plots suggest several Bayesian transitions, including transitions among the 4-gon variants and from 5 to 5+.The supplied figures describe these transitions as suggestions rather than definitive findings.Source: Figure D.4 · p. 26, Figure D.5 · p. 27

E Intuition for the local learning coefficient as complexity measure for c = 6

The appendix relates the local learning coefficient to parameter-space freedom and details the MCMC procedures used to compare predicted and observed phase occupancy. It reports a 5-to-6-gon switch at intermediate sample sizes, while noting classification and sampling limits.

E Intuition for the local learning coefficient as complexity measure for c = 6

  • For minimally singular 5-gons, fixing a positive bias removes three free parameters, making the 5+-gon learning coefficient larger than the 5-gon’s by 3/2.The authors use free-parameter counts as an intuition for complexity in these minimally singular cases.Source: E Intuition for the local learning… · p. 28
  • Large negative biases create extra parameter freedom in 4-gon variants, which the local learning coefficient classifies as lower complexity.For example, the 4−−-gon has fewer input representations than the 4−-gon, with added freedom in a dead activation region.Source: E Intuition for the local learning… · p. 28

F MCMC Experiments

  • Posterior samples are drawn with MCMC-NUTS under a standard multivariate Gaussian prior, using six chains and 3,000 retained samples per setting.Each combination of n and c is run for 10 PRNG seeds.Source: F MCMC Experiments · p. 29

F.1 Details of Theoretical Proportion Curves

  • The theoretical occupancy curves use f_α(n) = nL_α + λ_α log n + c_α, with theoretical λ values for 5, 5+, and 6 and estimated values for 4 and 4+.The extended plot includes sub-dominant phases; some local learning coefficients are estimates rather than exact values.Source: §F.1 · p. 29, Figure F.1 · p. 29

F.2 Verifying dominant phases for r = 2, c = 6

  • At n = 1000, the posterior is dominated by the 6-gon, followed by the 5-gon and then the 5+-gon; inspections place the dominant 5-to-6-gon switch at 600 ≤ n ≤ 700.The 5-gon cluster can include a nonzero vestigial leg while remaining in the 5-gon phase.Source: §F.2 · p. 30
  • Occupancy diagrams for 100 < n < 400 warrant caution because diffuse posterior samples and high-loss variants can affect classification and empirical frequencies.For n > 400, inspected samples are close to known critical points and unclassified samples are too infrequent to substantially affect dominant-phase frequencies.Source: §F.2 · p. 30

F.3 MCMC Health

  • MCMC health checks use loss traces and phase-type traces, but chains fail at sample sizes well above those reported as concentrated posteriors hinder escape from initialization.The appendix describes chains that remain in narrow loss bands or rarely leave their starting phase.Source: §F.3 · pp. 31–32

F.4 Theory and experiments for c = 4 and c = 5

For c=4 the 4-gon dominates at every sample size, while c=5 exhibits a 4-to-5 posterior transition. The local-coordinate appendix relates the loss to the TMS potential and describes chamber-based analysis, with extra work required at chamber boundaries.

F.4 Theory and experiments for c = 4 and c = 5

  • For c=4, the 4-gon dominates for all n; for c=5, posterior mass shifts from 4- to 5-gons, while 4+ never dominates despite lower energy.The c=4 theory predicts dominance because the 4-gon has zero loss and a 3-gon does not.Source: Figure F.6 · p. 34, §F.4 · p. 34, Figure F.5 · p. 34

G Potential in local coordinates

H.1 k-gons with negtive bias for k = c /∈4Z

This section characterizes k-gon critical points and their local learning coefficients, including where common solutions are found and extensions to higher dimensions. It also treats the degenerate k < c construction with extra zero-length dimensions and negative biases, noting that for k = 5, 6, 7, and 8 these points are minimally singular with a sum-of-squares local potential.

H.1 k-gons with negtive bias for k = c /∈4Z

  • For c = 5, 6, and 7, the negative-bias c-gons are non-degenerate critical points with local learning coefficients 7, 8.5, and 10, respectively; no common solution was found for 9 ≤ c ≤ 203 when c is not a multiple of 4.For the three cases, the coefficient follows (3c − 1)/2; the no-solution statement concerns the parameterized polynomial system.Source: §H.1 · p. 46

H.2 k-gons with negative bias for k = c ∈4Z

  • For c = 8, the 8-gon is a non-degenerate critical point with local learning coefficient 11.5; the text also reports common solutions at n = 12 and for every 1 ≤ s ≤ 50.At n = 12, two common solutions are reported; the 8-gon has coordinates with equal lengths approximately 1.55041 and equal biases approximately −1.29119.Source: §H.2 · p. 49

H.3 k-gons with negative bias for k < c

  • When a k-gon is a critical point for c = k under the theorem’s conditions, it remains critical for any c > k after adding zero lengths and negative biases for the extra dimensions.The theorem specifies equal lengths and biases among the k active coordinates, with the remaining lengths zero and biases negative.Source: §H.3 · p. 50
  • Although several chambers meet at the 5-gon when n = 6, the potential is analytic in an open neighborhood of that point.The proof establishes a single explicit local form for the potential around the 5-gon.Source: §H.3 · p. 52
  • Given a critical k-gon for c = k, the corollary extends it to c > k by setting the extra lengths to zero and their biases negative.The first k lengths and biases retain the equal-coordinate k-gon form; the corollary states that the extended point has an open neighborhood with the specified local potential form.Source: §H.3 · p. 52
  • For c > k and k = 5, 6, 7, or 8, the k-gon is degenerate but minimally singular, with the potential locally a sum of squares with positive coefficients.The local learning coefficients and losses for these k-gons are tabulated in Table H.2.Source: §H.3 · p. 53, Table H.2 · p. 54

I Including positive biases

For the positive-bias extensions, the paper establishes criticality for kσ+-gons under a stated condition on the corresponding k-gon, while leaving their minimal singularity unknown for c>k. For 4-gons, criticality can coexist with direction-dependent local learning coefficients.

I Including positive biases

  • For c=6, the 5+-gon is critical with λ=8.5; more generally, kσ+-gons are critical under the stated k-gon condition, though their minimal singularity is unknown for c>k.The authors describe a Hessian-based test for minimal singularity, but do not establish it for general kσ+-gons with c>k.Source: I Including positive biases · pp. 57–58

J 4-gons

  • At 4-gons, the TMS potential is not analytic and its directional Hessian depends on the approach direction.Source: J 4-gons · p. 59

J.1.1 Standard 4-gon

  • The standard 4-gon is a critical point with local learning coefficients {4, 4.5, 5, 5.5} across subspaces approaching it.Source: §J.1.1 · p. 60

J.1.2 4ϕ−-gon

Several 4-gon configurations are critical points of the TMS potential. For 4ϕ−-gons, the potential and local learning coefficients can vary by the configuration and approach subspace.

J.1.2 4ϕ−-gon

  • For ϕ=4, the 4ϕ−-gon’s local potential is zero, making it a critical point with local learning coefficient 0.Source: §J.1.2 · p. 61
  • For ϕ=2, reported local-learning-coefficient values are {2, 2.5, 3, 3.5}, with different directional Hessians across approaches.Source: §J.1.2 · p. 61
  • For ϕ=3, reported local-learning-coefficient values are {1, 1.5, 2, 2.5}; separately, {2.5, 3, 3.5, 4} are reported for B−={i, i+2}, where i=1, 2, 3, 4.Source: §J.1.2 · p. 61
  • The standard, 4ϕ−, 4σ+, and mixed 4σ+,ϕ− configurations are critical points of the TMS potential.Source: §J.1.2 · pp. 62–64

K Details of local learning coefficient estimation

The estimated local learning coefficients generally follow theoretical ordering, with closer agreement for low-loss critical points. Estimates for higher-loss points can be disrupted when SGLD chains escape to lower-loss states.

K Details of local learning coefficient estimation

  • The ordering of estimated λ̂ across critical points matches theoretical predictions, and low-loss 6, 5, and 5+-gons have estimates close to theory.Source: K Details of local learning… · p. 65
  • For higher-loss points, a misconfigured SGLD step size can drive chains into lower-loss phases, producing negative λ̂; smaller steps help but require more sampling.Source: K Details of local learning… · p. 65
  • Estimates used n=5000, γ=0.1, 10,000 steps, and 10 chains per critical point; chains with more than 5% below-point loss were discarded.Source: K Details of local learning… · p. 65
  • Table K.1 reports λ̂, viable-chain standard deviations, and selected SGLD step sizes for known critical points with r=2 and c=6.Source: Table K.1 · p. 66
Loading 2310.06301v1…