Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Sai Niranjan Ramachandran; Suvrit Sra

Source-linked AI summary

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Sai Niranjan Ramachandran, Suvrit Sra

arXiv:2609.02373v1Sep 2, 2026cs.LGcond-mat.dis-nncond-mat.stat-mechcs.AI

TL;DR

The paper asks how SGD reaches simpler invariant sets, then models stochastic gradient flow as symmetry-induced percolation. It finds discrete block-merges, variance spikes, and a scaling cascade extending to Adam and AdamW under heavy-tailed noise.

Problem
The dynamics by which SGD steers networks onto simpler invariant sets remain poorly understood, limiting mechanistic explanations of anomalous training behaviors.
Method
The paper models stochastic gradient flow as a percolation process whose architectural symmetries produce discrete simultaneous subnetwork merges.
Results
The framework identifies variance spikes at structural transitions, proves discrete scale invariance, and extends the trapping mechanism and cascade to Adam and AdamW.
Takeaways & Limitations
Percolation topology and relative variance provide a model-based way to characterize optimization collapse and its discrete scaling cascade.
Takeaways & Limitations
The Adam and AdamW extension assumes heavy-tailed gradient noise and non-degenerate coordinate-wise preconditioners.

Abstract

from arXiv · show

We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.

1 Introduction

The paper addresses how SGD steers networks toward simpler invariant sets by modeling optimization as symmetry-induced percolation. It links discrete structural merges to variance-based transitions and extends the framework to adaptive optimizers.

SGD can collapse networks onto invariant sets behaving like simpler subnetworks, but the dynamics reaching those sets remain poorly understood.
The framework represents stochastic gradient flow as percolation, where isolated components link and fuse as optimization progresses.
Architectural symmetries force simultaneous block-merges rather than continuous one-edge attachment, and the paper determines when this discontinuity survives at large network size.
The trapping mechanism and cascade structure extend to Adam and AdamW under an explicit heavy-tailed noise model.
Relative variance across training trajectories isolates discrete microtransitions, while the framework formalizes SGD dynamics and proves discrete scale invariance.

2 Stochastic Collapse in SGD

This section models SGD as stochastic gradient flow and characterizes how architectural invariant sets trap trajectories. Under affine symmetry, the continuous SGF preserves discrete SGD invariance and suppresses transverse escape.

SGD is approximated continuously by an Itô SDE separating full-batch gradient drift from zero-mean, position-dependent mini-batch noise.
Architectural symmetries create invariant parameter regions, and affine invariance in discrete SGD transfers exactly to continuous SGF under bounded Lipschitz sample gradients.
The continuous-process definition requires trajectories initialized in an invariant set to remain there with probability one.
Stochastic attractivity arises when mini-batch variance decays near an invariant set, producing an inward pull that can trap trajectories.

3 Topological Dynamics of SGF via Attractor Equivalence

The paper formalizes synchronized subnetwork trajectories as attractor equivalence classes and projects their merges and splits onto a Reeb graph. This topology yields a percolation-like description of condensation and fragmentation.

A block-diagonal covariance assumption treats symmetry-decoupled subnetworks as conditionally independent before they collapse together.
Theorem 3.3 models stopped transverse distance as a non-negative local supermartingale, bounding subnetwork escape from an attractive invariant-set basin.
Probabilistic non-escape bounds trajectory escape from an epsilon-neighborhood by the initial transverse distance.
Attractor Equivalence binds synchronized path measures, and Metastable Equivalence makes this relation a time-parameterized equivalence class while escape remains untriggered.
The Reeb graph represents condensation as node merging and fragmentation as branch splitting when stochastic variance breaches the local basin.
Each merge or split rewires the global covariance matrix by fusing or severing off-diagonal covariance blocks.

4 Percolative Collapse and Discrete Scale Invariance

Symmetry-induced block-merges create discontinuous percolative collapse rather than continuous edge growth. Relative-variance spikes reveal these transitions, which follow a discrete scale cascade and forecast global collapse.

Approximate Q-symmetry restricts deterministic drift to an affine invariant set and decouples transverse distance into curvature-free diffusion.
Temporal renormalization converts the fluctuating Reeb graph into a monotonic percolation graph by integrating out fast microscopic noise.
The structural order parameter tracks the fractional size of the largest connected component.
A jump ∆O(p) = (n −1)i/N is thermodynamically discontinuous precisely when merging components are macroscopic, i = Θ(N); microscopic merges vanish asymptotically.
Relative variance Rv(p) diverges in proportion to (∆O)2 at structural microtransitions before the global connectivity threshold pc.
Discrete scale invariance uses preferred magnification factors λ(n) = nσ, and symmetric hyper-condensation produces a geometric cascade forecasting global collapse pc.

5 Extension to Adam and AdamW

The paper extends its SGD trapping and discrete-scale-invariance framework to Adam and AdamW under explicit heavy-tailed-noise and macroscopic-block assumptions. Under these conditions, adaptive preconditioning preserves the relevant affine geometry and cascade scaling.

Extension scope: The trapping mechanism and DSI cascade extend from SGD to Adam and AdamW under explicit conditions.The extension is developed with full derivations in Appendix D.
Symmetry: Coordinate squaring narrows the admissible symmetry group to coordinate permutations, which already contains the neuron-permutation symmetry used in the SGD theory.The corresponding joint invariant set includes parameters, momentum, and second-moment states.
Noise model: Heavy-tailed gradient noise is modeled with a finite p-th moment for p ∈ (1, 2], allowing infinite variance when p < 2.The analysis truncates the noise at a time-dependent threshold, introducing a controlled bias.
Noise filtering: Adam’s momentum and second-moment recursions are treated as exponential moving-average filters with memory scales τ1 = 1/(1 − β1) and τ2 = 1/(1 − β2).The second-moment transform is obtained through the characteristic function rather than by squaring the transformed gradient directly.
Conditional trapping: Under non-degeneracy, low-correlation, and macroscopic-block assumptions, the stopped transverse process remains a non-negative local supermartingale for Adam and AdamW.The assumptions require ˆvi(θt) ≥ vmin > 0 and n(N)/N → c ∈ (0, 1].
Cascade result: Adam- and AdamW-trained trajectories satisfy generalized DSI with the same magnification factor λ(n) = n^σ as the SGD theorem.The scalar preconditioning factor cancels in the ratio defining λ(n), while macroscopic block scaling preserves the thermodynamic-limit discontinuity.

6 Empirical Validation

Experiments test percolative collapse through a macroscopic order parameter and detrended variance fluctuations across controlled, grokking, tabular, and vision settings. The results reproduce the predicted scaling cascade and its reversibility, while associating a three-peak cascade with delayed generalization.

Experimental setup: Percolative collapse is evaluated using a macroscopic order parameter O and its detrended variance fluctuations Rv across training-seed ensembles.The ensemble design targets the bimodal-coexistence requirement of Theorem 4.5.
Controlled toy setting: λ ≈2.00 appears in both constrained K = 3 and unconstrained K = 6 SGD toy models through localized Rv divergences.A task shift at t = 3000 produces a reactive fragmentation peak, confirming the predicted reversibility.
High-dimensional evaluation: Spectral Effective Rank replaces distance-based clustering when testing collapse in unconstrained, high-dimensional networks.
Grokking: λ = 2.11 and a phase-randomized spectral-null FPR of 0.1% accompany a three-peak DSI cascade immediately preceding the Transformer’s performance spike.The paper does not claim that the cascade is the sole driver of the transition.
Benchmark generalization: The same cascade structure appears across UCI tabular and vision benchmarks, with fractional scaling factors consistent with pairwise or higher-order merges.

7 Related Work

The paper positions its approach against work on low-rank dynamics, percolation-based training analyses, spectral dimensionality measures, and condensation. Its distinguishing feature is a symmetry-driven renormalized Reeb graph for transient collapse.

Stochastic dynamics and implicit bias: Prior work models SGD as driving networks toward low-rank manifolds through anomalous diffusion and glassy dynamics.This paper instead tracks the transient path to those manifolds through network percolation on a continuous Reeb graph.
Graph and percolation perspectives: Recent percolation studies examine connectivity under dropout or synaptic invariants that forecast performance.The paper’s percolation graph addresses a different structural object.
Dimensionality measures: Spectral Effective Rank measures soft dimensionality collapse from the exponentiated Shannon entropy of singular values.It generalizes hard exact-zero collapse to settings where weights rarely vanish exactly.
Empirical context: Figure 2 places the paper’s empirical cascade alongside Transformer grokking, UCI Heart Disease, and FMNIST evaluations.
Paper’s distinction: The paper’s graph is constructed from a renormalized Reeb graph tracking collapse into shared invariant sets, with edges driven by architectural symmetry.This differs from graphs based on random deletion or correlation.
Neuron and weight condensation: Condensation work provides a complementary lens focused on initialization scale rather than symmetry.The paper’s mechanism targets explicit transient dynamics.

8 Limitations, Conclusion and Future Work

The paper formalizes SGF as a continuous counterpart to SGD, proves invariant-set transfer and probabilistic trapping, and derives symmetry-induced discontinuities. It confirms discrete scale invariance theoretically while identifying extensions and practical-width persistence as future work.

Conclusion and future work: The paper proves DSI for the symmetry-induced percolation process, confirms it exactly in toy models, and finds consistent cascades across tabular, vision, and grokking settings.Future work includes testing persistence at practical network widths and extending the framework to curved invariant manifolds and extreme hyperparameters.
Formalization: The framework formalizes the relationship between discrete SGD and continuous SGF and defines invariant sets for the continuous process.Affine invariant sets in SGD strictly translate to SGF under the stated regularity conditions.
SGF construction: SGD noise is decomposed into zero-mean per-sample deviations, yielding an SGF diffusion matrix factored into magnitude Dm and shape Ds(θ).Hyperparameters control Dm, while architecture, data, and loss topology determine Ds(θ).
Trapping: For stochastically attractive invariant sets, the stopped transverse process is a non-negative local supermartingale and escape probability is bounded by initial transverse distance.This establishes probabilistic non-escape within the local basin.

B.2 Attractor Equivalence

The paper relaxes ideal tail equivalence into a metastable, basin-conditioned relation and uses it to construct a dynamic topological network whose edges encode subnetwork coupling and splitting.

t-Tail equivalence requires subnetworks to share an invariant manifold and assign identical probabilities to every future event in the tail σ-algebra.
Finite-time noise makes strict tail equality difficult, motivating metastable equivalence conditioned on trajectories remaining near a shared invariant manifold.
Attractor link probability measures whether both subnetworks remain inside the local basin from time t onward.
As transverse drift vanishes, maximal survival bounds drive escape probabilities toward zero and joint link probability toward one.
Attractor equivalence defines permanent coupling through vanishing transverse drift, while metastable equivalence forms an equivalence class bounded by the local basin.
The ϵ-fixed network uses attractor equivalence as adjacency, and its continuous Reeb graph records condensation into merged nodes and fragmentation into separated branches.

C.1 Approximate Q-Symmetry, Affine Trapping, and Transverse Diffusion

Permutation symmetries generate flat affine invariant sets that trap stochastic trajectories and reduce transverse dynamics to one dimension. Timescale separation then yields irreversible percolation, with block merges producing finite-size discontinuities and variance spikes.

Permutation invariance among identical components generates flat affine invariant sets that partition parameter space and can trap learning trajectories.
The transverse distance Y_t reduces exactly to a one-dimensional diffusion without curvature drift, isolating escape and re-entry dynamics.
Curved invariant manifolds would introduce mean-curvature drift, couple transverse and tangential motion, and require geometric potential terms for edge-rate modeling.
On the fast timescale, stationary current forces inward and outward boundary fluxes to cancel, so the net topological change is zero.
Under τfast ≪ τslow, renormalization removes transient fluctuations and produces a macroscopic percolation graph governed by irreversible attractor equivalence.
The order parameter advances monotonically from 0 to 1 as symmetric subnetworks collapse onto the invariant set.
For merges of size i, the jump is ∆O(p) = (n − 1)i/N; it survives as N →∞ when i = Θ(N), but vanishes when i = O(1).
Macroscopic block merges create divergent relative-variance peaks, while symmetric hyper-condensation restricts growth to C1 → nC1 and generates a geometric transition cascade.

D.3 Trapping and the DSI Cascade

The analysis bounds Adam’s preconditioner and shows that stochastic trapping produces a DSI cascade under explicit block-scaling assumptions. The theory predicts λ(n)=n^σ, with pairwise mean-field merges giving λ=2.

Adam preconditioner: Corollary D.20 bounds the tracking error of each coordinate’s second-moment estimate using deterministic drift and filtered-noise terms.The bound relies on Lipschitz control, non-degeneracy, and a stopping-time event.
Adam preconditioner: Theorem D.23 reduces Adam and AdamW dynamics to a vanilla-SGD-like diffusion with a residual term and positive scalar rescaling.The reduction holds under the stated assumptions and with high probability on the block timescale.
DSI cascade: Under macroscopic block scaling, Adam- or AdamW-trained trajectories admit generalized DSI with the same magnification factor λ(n)=n^σ.The merging block must remain macroscopic relative to network size for the cascade to persist.
Reversibility: The theoretical threshold makes collapse reversible: reduced noise and increased signal curvature can render the invariant manifold repulsive and trigger fragmentation.The reverse transition is tied to violating the trapping condition.
DSI cascade: For pairwise identical merges under mean-field percolation, the predicted scaling is λ=2, so divergence times satisfy t_k+1=2t_k.The factor follows from n=2 and σ=1.
Empirical validation: The kinematic experiment shows discrete order-parameter jumps and relative-variance divergences at topological merge points, with binding at t1=700 and tc=1400.These transition times yield an empirical log-linear scaling progression.

E.4 Empirical Simulation (K=6) with Task Shift and Reverse Transition

The K=6 SGD simulation uses a high-learning-rate condensation phase followed by a task shift that violates the trapping threshold. Variance peaks mark both forward collapse and reverse fragmentation.

Condensation phase: During condensation, high learning rate injects multiplicative noise intended to satisfy the stochastic collapse condition.The experiment separates condensation from the task-shift phase at t=3000.
Condensation phase: High stochastic diffusion drives independent parameters into shared, lower-dimensional invariant subspaces.Different ensemble realizations produce shifted merge thresholds and a multi-phase variance cascade.
Variance cascade: Sequential relative-variance peaks occur within the high-learning-rate regime during condensation.These peaks are interpreted as precursors of the structural merges.
Reverse transition: At t=3000, reduced learning rate and a new high-frequency target violate the trapping threshold, making the invariant manifold stochastically repulsive.Previously bound parameters then fragment to represent the new target dimensionality.
Reverse transition: The task shift produces an immediate relative-variance divergence peak associated with discontinuous fragmentation.This is the reverse counterpart to the earlier condensation transition.

F.1 Methodological Extensions for Empirical Regimes

For unconstrained empirical networks, the paper replaces hard component counting with spectral Effective Rank and detrended log-variance. This yields fractional DSI factors linked to the spectral variance mass involved in a merge.

Effective Rank: Effective Rank maps macroscopic dimensionality collapse to spectral concentration rather than weights becoming exactly zero.The metric is computed from the singular-value spectrum of the target weight matrix.
Signal processing: Detrended log-variance isolates structural variance spikes from the global heteroskedastic decay of SGD noise.The pipeline uses log(Var[O(t)]) and subtracts a global linear secant before smoothing.
Evaluation setup: The empirical analysis aggregates 20 initializations for tabular datasets and 15 for vision and grokking datasets.Experiments were run on a Tesla T4 GPU and required approximately one day of compute.
Spectral relaxation: Theorem F.1 gives the fractional scaling law λ_eff=nm^σ for an n-body merge carrying fractional spectral variance mass m.This relaxes the rigid integer factor obtained from idealized topological counting.
Empirical scaling: λ=2.11 for Transformer grokking exceeds the pairwise upper bound and, under a 3-body merge, implies m≈0.68.The paper presents the inferred variance mass as a hypothesis about dismantled algorithmic circuits.
Empirical scaling: λ=1.28 for UCI Abalone implies m≈0.36 under the pairwise-collapse hypothesis.The inferred mass is the active spectral variance attributed to the transition.
Empirical scaling: λ=1.71 for Heart Disease and λ=1.57 for FashionMNIST imply m≈0.77 and m≈0.65, respectively, under pairwise collapse.Both inferences use the relation 2m=λ.

F.2 UCI Tabular Datasets

Across UCI tabular datasets, Effective Rank decays smoothly while detrended variance reveals discrete DSI microtransitions. Null testing distinguishes a supported Heart Disease signal from a German Credit false positive.

Dataset-wide pattern: Shallow MLPs trained on UCI datasets show smooth continuous decay in Effective Rank and training loss.Detrended variance nevertheless exposes discrete DSI microtransitions.
Null-model evaluation: Heart Disease yields a statistically significant mapping of internal topological constraints with FPR=4.9%.The result is reported as evidence for the extracted structural signal.
Null-model evaluation: German Credit produces a 4-peak sequence with λ=1.99 and R2=0.86, but spectral null testing gives FPR=80.2%.The paper treats this as a critical negative control.
Pipeline necessity: The German Credit negative result shows that standard optimization noise can mimic geometric scaling artifacts without the signal-processing pipeline.This establishes the need for detrending and null-model evaluation.
Evaluation summary: Table 1 aggregates empirical DSI fits and statistical null-model evaluations across UCI tabular datasets using 20 independent initializations.The table is the dataset-level summary of these evaluations.

F.3 Geometric Manifolds, Vision, and Algorithmic Grokking

Across geometric manifolds, vision benchmarks, and modular-arithmetic grokking, variance fluctuations track discrete structural condensation, with cascade strength varying by task and architecture.

Cross-domain evaluation: The evaluation spans geometric manifolds, vision benchmarks, and modular-arithmetic grokking to test whether scaling laws extend beyond tabular data and shallow MLPs.The modular-arithmetic experiment uses a Transformer optimized with AdamW and weight decay.
Tabular comparisons: German Credit provides a negative control: its 4-peak cascade has λ = 1.99 and R2 = 0.86 but a spectral null FPR of 80.2%.Heart Disease instead shows dimensionality collapse mirrored by a sequential 4-peak variance divergence with λ = 1.71, R2 = 0.98, and FPR = 4.9%.
Algorithmic Grokking: The modular-arithmetic Transformer reaches effective rank 10.93 ± 0.99 and exhibits a perfectly log-linear 3-peak DSI cascade with λ = 2.11, R2 = 1.00, and FPR = 0.1%.Across 15 initializations, final train loss is 0.005 ± 0.004 and validation loss is 0.030 ± 0.033.
Vision: FashionMNIST yields a 3-peak cascade with λ = 1.57, R2 = 1.00, and spectral null FPR = 10.7%, whereas MNIST shows a 2-peak sequence without a full extended cascade.Both vision settings associate variance fluctuations with discretized capacity reduction or early-stage topological condensation.
Geometric Manifolds: Moons and Swiss Roll each show a primary structural microtransition while MLPs parameterize nonlinear 2D and 3D spatial embeddings.The reported divergences mark spatial alignment during early-stage optimization.