PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable initialization without the CLT

Simon Kuang, Kyle Chickering, Xinfan Lin

arXiv:2609.30633v1cs.LG

TL;DR

Stable initialization is important because neural networks are sensitive to initial weights, while conventional theory relies on a Central Limit Theorem approximation with error. The paper uses sine’s periodicity to construct uniform-phase initialization and reports exact stability guarantees and zero-shot input/output tuning on par with grid search.

  • Problem

    Conventional stable-initialization theory uses a Central Limit Theorem approximation that incurs O(N^-1) error, while narrow networks can be appreciably non-Gaussian at initialization.

    Source: §1 · p. 2

  • Method

    Uniform-phase initialization uses large biases to distribute sine preactivations uniformly around the torus, making preactivations independent and identically distributed.

    Source: §1.1 · p. 2, §5 · p. 9

  • Results

    The initialization gives exact stability guarantees for every fixed input, with prescribed values attained exactly and guarantees applying at any width and depth.

    Source: §1.1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Data-driven input/output scale tuning achieves zero-shot performance on par with a grid search over prior hyperparameterizations.

    Source: §5 · p. 10

    The conventional Central Limit Theorem-based theory incurs O(N^-1) error, and narrow networks are appreciably non-Gaussian at initialization.

    Source: §1 · p. 2

Abstract

from arXiv · show

Successful training of deep neural networks is highly dependent on the distribution of the initial weights. If the weights are too large, network training blows up; if they are too small, the model fails to learn features. Stable initialization is the optimal moderation between these two extremes. The conventional theory of random networks uses the Central Limit Theorem to control inter-neuron dependencies, which introduces distributional approximation error and coupling between layers. For networks with sine activations, we derive the uniform-phase initialization, which obviates distributional approximation and fully decouples the layers. Ours is the first work to use the sine function's periodic symmetry. Models trained with the uniform-phase initialization outperform the state of the art in neural representation tasks like image and audio fitting. We find that our untuned models are competitive with the best-tuned baselines from previous work and support $\mu$P width scaling.

1 INTRODUCTION

Stable initialization seeks to prevent gradient instability as networks grow, but conventional analyses use a Central Limit Theorem approximation that is inaccurate for narrow networks. This work uses sine’s periodicity to create uniform-phase initialization, giving exact stability guarantees and data-driven input/output tuning.

1 INTRODUCTION

  • Conventional stable-initialization theory controls preactivations and gradients through a layerwise recursion closed by a large-width Central Limit Theorem approximation.That approximation incurs O(N^-1) error, and narrow networks can be appreciably non-Gaussian at initialization.Source: §1 · p. 2

1.3 RELATED WORK

  • Prior sine-network initialization work used moments, Taylor expansions, and range, but did not exploit sine’s periodicity.Earlier work includes attempts to control preactivations and gradients, while other approaches target frequency spectra or signed-distance functions.Source: §1 · p. 2, §1.3 · pp. 3–4

1.1 SUMMARY OF OUR METHOD

  • The uniform-phase initialization yields exact stability guarantees at any width and depth by making preactivations independent and identically distributed.Large biases spread preactivations uniformly around the torus, regardless of the previous layer’s location and scale.Source: §1.1 · pp. 2–3

1.2 CONTRIBUTIONS

  • The method combines periodicity-based hidden-layer initialization with exact moment matching for zero-shot input/output tuning, and is reported to meet or exceed prior work on most examples.The tuning adapts to target resolution and output statistics.Source: §1.2 · p. 3

2 THEORY OF THE UNIFORM-PHASE INITIALIZATION FOR HIDDEN

Uniform-phase biases make sinusoidal hidden-layer preactivations uniform modulo 2π, enabling independent factorization and exact layerwise gradient-moment guarantees.

2 THEORY OF THE UNIFORM-PHASE INITIALIZATION FOR HIDDEN

  • The hidden map applies each layer as zℓ = Wℓxℓ−1 + bℓ, followed by xℓ = sin(zℓ).Source: §2 · p. 4
  • Uniformly distributed phase modulo 2π makes the final preactivation uniform for any fixed preceding input, allowing its bias average to factor out.This uniform-phase step is the key to the independence argument.Source: §2 · p. 4
  • Circular symmetry gives zero mean cosine and equal expected squared sine and cosine factors, with their squared values summing to one.Source: §2 · p. 5
  • The theorem supplies forward recursions in t and backward recursions in s for the hidden-layer quantities.Source: §2 · p. 5
  • Between any two layers, the Jacobian is isotropic and has average squared singular value 1, so early loss gradients neither grow nor decay on average backward.The network’s range and slope properties are also independent of width and depth.Source: §2 · p. 6

3 MOMENT MATCHING FOR INPUT AND OUTPUT LAYERS

The input and output layers are calibrated by matching target moments, while a uniform input bias provides translation invariance and removes the need to center data.

3 MOMENT MATCHING FOR INPUT AND OUTPUT LAYERS

  • A uniform input bias establishes translation invariance and eliminates the need to center the data.Source: §3 · p. 6
  • The method moment-matches target mean μ, covariance Σ, and structure tensor Ω when initializing the input weights, output weights, and output bias.Source: §3 · p. 6
  • Because the initialization is translation invariant, moments may be evaluated at any particular input or over a representative input distribution.Source: §3 · p. 6
  • Using Ω to calibrate input scale in the physical domain nondimensionalizes the inputs and outputs around the hidden network.Source: §3 · p. 6

4 NUMERICAL EXPERIMENTS

The experiments test stable initialization across image and audio fitting, narrow networks, and width scaling. The moment-matched uniform-phase approach performs competitively without hand-tuned frequencies and is evaluated alongside μP learning-rate scaling.

4 NUMERICAL EXPERIMENTS

  • The experiments test stable initialization as the hypothesis that a stably initialized network performs better after thousands of epochs under the same training operator.The study evaluates Cameraman fitting on a training grid and a larger test grid.Source: §4 · p. 6, §4.1 · p. 6

4.1 IMAGE FITTING

  • The uniform-phase, moment-matched initialization achieves the lowest evaluation MSE on Cameraman by a small margin.The top two methods preserve fine details without excessive noise; the evaluation uses a 512 × 512 test grid after fitting on 128 × 128.Source: Figure 1 · p. 7, §4.1 · p. 7

4.2 AUDIO FITTING

  • 29.33 dB: the moment-matched SIREN’s audio SNR exceeds CVP26’s by nearly 20 dB without a hand-tuned frequency, although SM20 reaches 31.15 dB.Non-sinusoidal networks collapse to near-zero output in this experiment.Source: §4.2 · p. 7

4.3 NARROW NETWORKS

  • 10.28 dB: the moment-matched SIREN is best across all reported metrics, widening its gap to CVP26 to 6.8 dB in the narrow-network experiment.Source: §4.3 · p. 7
  • On grayscale and RGB images, moment matching achieves the lowest test MSE on 5 of 8 images and remains competitive on the other three.On Brick, Gravel, and Hubble, it loses to the best-tuned baseline by 19%, 12%, and 1.5%, respectively; it is no worse than mistuning ω0 by 20%.Source: §4.3 · p. 8
  • For all CIELAB images, the initialization achieves the best fit, with improvement factors of 3.0 to 5.2 on images other than Hubble.The improvement on Hubble is slight.Source: §4.3 · p. 9

4.5 WIDTH SCALING USING µP: INITIALIZATION AND LEARNING RATES

  • Under both μP-informed scalings, loss deepens with width, and both the optimal learning rate and the phase transition shift right; optimal hyperparameter transfer was not observed.Both μP scalings yield lower loss with increasing width at every learning rate.Source: §4.5 · p. 9

5 CONCLUSION

The paper presents uniform-phase hidden-layer initialization and moment matching for input-output scales as a simplified approach with guarantees across widths and depths. Its conclusion is that zero-shot tuning can match prior grid searches, while sine may have uses beyond neural representation.

5 CONCLUSION

  • The hidden-layer method makes preactivations independent and identically distributed, enabling exact gradient-moment calculations and guarantees for all widths and depths.Source: §5 · p. 9
  • Data-derived covariance and structure tensor calibrate input and output scales in the physical domain, producing zero-shot tuning on par with prior grid searches.Source: §5 · p. 10
  • The authors suggest that sine’s uniform-phase property may prove fruitful beyond neural representation.Source: §5 · p. 10

A COMPARISON TO CONVENTIONAL STABLE INITIALIZATION OF HIDDEN LAYERS

  • The appendix comparison uses Combette et al. (2026) as a state-of-the-art implementation of conventional stable initialization for hidden layers.It describes two significant derivations in that comparison.Source: A COMPARISON TO CONVENTIONAL STABLE… · p. 14

A.1 PREACTIVATION DISTRIBUTION

Conventional preactivation analysis relies on Normal approximations and limits in width and depth; uniform phase makes wrapped preactivations exactly uniform and decouples layers. The same setup supports a layer-Jacobian result without those asymptotics.

A.1 PREACTIVATION DISTRIBUTION

  • Prior analysis requires N→∞ for the Central Limit Theorem and L→∞ to wash out the embedding layer’s transient effect.The resulting moment recursion has an attractive fixed point.Source: §A.1 · p. 15
  • Uniformly distributed hidden biases make each wrapped preactivation uniform on the N-torus, regardless of the preceding layer’s distribution.This exact construction removes the need for recursive analysis or limits in width and depth.Source: §A.1 · p. 15

A.2 ACTIVATION GRADIENT DISTRIBUTION

  • Under the uniform wrapped-preactivation setup, the layer-Jacobian entries have zero mean and a shared variance, without requiring large-width asymptotics.The result assumes zero-mean weights with variance c_w^2/(3N); the supplied passage does not preserve the resulting Jacobian-variance expression.Source: §A.2 · p. 16

A.3 WHAT GRADIENT CRITERION?

The section contrasts stabilizing individual Jacobian entries with stabilizing the Gramian. Uniform-phase initialization removes asymptotics from the Jacobian second-moment calculation, while translating it into loss-gradient predictions requires an independence assumption on the loss cotangent.

A.3 WHAT GRADIENT CRITERION?

  • Combette et al. stabilize layer-Jacobian entries, whereas this work stabilizes the Jacobian Gramian.The prior scaling analysis is described as plausible but not fully rigorous, relying on an unjustified independence heuristic, a double limit, and an unknown constant.Source: §A.3 · p. 16
  • Uniform-phase initialization removes asymptotic limits from the Jacobian second-moment calculation, but predicting loss-gradient scaling requires an additional assumption on the loss cotangent.The remaining approximation treats the cotangent as approximately independent of hidden-layer initialization, an early-training heuristic; for general losses it depends on the initialized network.Source: §A.3 · pp. 16–17
  • The uniform-phase result expresses parameter and input gradients in terms of the output loss cotangent and hidden-layer quantities.The stated setup defines g as the gradient of the loss with respect to the hidden-map output.Source: §A.3 · p. 16

A.4 SPECTRAL DIAGNOSTICS

The spectral diagnostic compares uniform-phase initialization with prior schemes across depth, revealing distinct stabilization targets and mechanisms. The supplement notes that µP’s last-layer scaling changes output magnitude relative to moment matching.

A.4 SPECTRAL DIAGNOSTICS

  • Uniform-phase spectra broaden with depth: the largest singular values grow while the smallest approach zero, with average squared singular value fixed at unity.Figure 3 compares the full end-to-end Jacobian spectra for four initialization schemes across depth.Source: §A.4 · p. 17, Figure 3 · p. 18
  • CVP26 σa=0 preserves the largest singular value, whereas uniform phase stabilizes the average squared singular value; the methods target different amplification criteria.Under CVP26 σa=0, average squared singular value decays even as peak amplification holds steady.Source: §A.4 · p. 17
  • CVP26 concentrates preactivations on sine’s steep region, while uniform phase averages across the full sine; SM20’s smaller biases accompany gradient blowup.The comparison holds the weight distribution constant between SM20 and uniform-phase initialization.Source: §A.4 · p. 17

B SUPPLEMENT TO §4.5

  • µP scales last-layer weights by N^-1 rather than moment matching’s N^-1/2, causing network output to decay as N^-1/2.The supplement notes that representation learning can be sensitive to output scaling despite the last-layer-only problem being convex.Source: B SUPPLEMENT TO §4.5 · p. 18

C TASKS AND EXPERIMENTAL METHODS

The methods specify shared fitting protocols, task-specific image and audio datasets, hyperparameter sweeps, and a Jacobian-spectrum diagnostic. Uniform-phase initialization is constructed from training-field statistics, while baselines follow their reference implementations.

Common protocol and metrics

  • Fitting uses full-batch MSE and Adam without weight decay, gradient clipping, or a learning-rate schedule; evaluation arrays are complete grids or full signals, not independent samples.The protocol uses 32-bit model parameters and arrays, with moment estimates accumulated in 64-bit precision.Source: C TASKS AND EXPERIMENTAL METHODS · p. 18

Initialization implementations

  • Uniform-phase MM uses training-field sample mean and covariance, Sobel-estimated spatial Jacobians, Gaussian input and hidden weights, and uniform input phases and hidden biases.Its output bias is the sample mean, and output-matrix columns are sampled using the estimated covariance.Source: C TASKS AND EXPERIMENTAL METHODS · p. 18
  • SM20 and CVP26 implementations follow their reference releases, while Tanh positional encoding, ReLU, SiLU, and GeLU use their specified standard initialization schemes.The MLP baselines match the compared SIRENs in width and depth.Source: C TASKS AND EXPERIMENTAL METHODS · pp. 18–19

Cameraman image fitting and narrow networks

  • Cameraman fitting trains on 16,384 downsampled coordinates and evaluates on 262,144 native-resolution coordinates; the narrow-network variant changes width from 256 to 16.The standard model uses depth 11 and 3,000 Adam steps.Source: C TASKS AND EXPERIMENTAL METHODS · p. 19

Audio fitting

  • Audio fitting uses the complete 308,207-sample signal for optimization and evaluation, with depth 5, width 256, and 5,000 full-batch Adam steps.No held-out or subsampled split is used.Source: C TASKS AND EXPERIMENTAL METHODS · p. 19

Input-frequency sweep

  • Input-frequency sweeps compare three prior initializations over 21 frequencies, while uniform-phase MM is initialized and trained once per image.Runs use four grayscale and four color images, with training and evaluation grids up to 128 and 512 pixels per dimension.Source: C TASKS AND EXPERIMENTAL METHODS · p. 19

Width and learning-rate sweep

  • The width and learning-rate sweep evaluates widths 8–512 across learning rates, using time-averaged pre-update training MSE rather than final or test loss.The µP conditions scale hidden and output learning rates by N0/N, with an additional output-weight scaling in µP (LR+LL).Source: C TASKS AND EXPERIMENTAL METHODS · p. 19

Jacobian-spectrum diagnostic

  • The Jacobian diagnostic measures sorted hidden-to-hidden singular values at depths 4, 8, 16, and 32, averaging 50 spectra per initialization and depth.It uses width 256 and evaluates at 10 input points across five independently initialized networks.Source: C TASKS AND EXPERIMENTAL METHODS · pp. 19–20

D ADDITIONAL EXPERIMENTAL RESULTS

The additional experiments cover quantitative Cameraman results, narrow-network image fitting, Bach audio fitting, and µP learning-rate transfer.

D ADDITIONAL EXPERIMENTAL RESULTS

  • Table 1 reports Cameraman train MSE on the 128 × 128 grid and evaluation MSE and SNR on the 512 × 512 grid.Best values are bolded.Source: Table 1 · p. 20
  • Table 2 reports quantitative results for the narrow (N = 16) Cameraman experiment.Source: Table 2 · p. 20
  • Figure 4 presents image fitting with narrow networks (N = 16), using the layout of Figure 1.Source: Figure 4 · p. 20
  • Figure 5 compares ground truth and predictions across initializations for the Bach recording, with full-waveform and first-1000-sample views and panel SNRs.The full-waveform view envelopes all 308,207 samples.Source: Figure 5 · p. 21
  • Figure 6 plots geometrically time-averaged train MSE for Cameraman across initializations, three scaling schemes, and widths from N = 8 to N = 512.Stars mark each width’s best learning rate.Source: Figure 6 · p. 22

E MOMENT-MATCHED PARAMETERS AND BEST-ω0 TEST ERRORS

The section reports best-ω0 test-MSE sweeps for grayscale, RGB, and CIELAB images, and describes how CIELAB parameter estimates differ from RGB.

E MOMENT-MATCHED PARAMETERS AND BEST-ω0 TEST ERRORS

  • Table 3 reports the best test MSE per method for grayscale images, with optimal ω0 values shown in parentheses.Bold entries mark the best value per row; visual comparisons are in Figure 7.Source: Table 3 · p. 23
  • Table 4 reports the best test MSE per method for color images, with conventions matching Table 3.Visual comparisons are in Figure 8.Source: Table 4 · p. 23
  • Compared with RGB, CIELAB output covariance has larger and mixed-sign off-diagonal entries, while the structure tensor is nearly unchanged.The passage attributes these differences to CIELAB’s wider dynamic range and the decorrelating effect of its opponency axes, respectively.Source: E MOMENT-MATCHED PARAMETERS AND… · p. 24
  • Table 5 reports the best test MSE per method in CIELAB coordinates, with conventions matching Table 3.Visual comparisons are in Figure 9.Source: Table 5 · p. 24

F BEST-ω0 VISUAL COMPARISONS

The visual comparisons show best-ω0 fits for grayscale, RGB, and CIELAB images, arranged alongside ground truth and baseline fits.

F BEST-ω0 VISUAL COMPARISONS

  • Each figure compares four images across ground truth, uniform-phase MM, and best-tuned SM20 and CVP26 fits.Baseline fits use the Appendix E ω0 sweep; panels show test MSE and the baseline ω0.Source: F BEST-ω0 VISUAL COMPARISONS · p. 25
  • Figure 7 presents grayscale best-ω0 comparisons for Brick, Camera, Gravel, and Clock.Source: Figure 7 · p. 25
  • Figure 8 presents RGB best-ω0 comparisons for Hubble, Skin, Astronaut, and Coffee.Source: Figure 8 · p. 26
  • Figure 9 presents CIELAB best-ω0 comparisons for Hubble, Skin, Astronaut, and Coffee, displaying predictions converted to RGB while computing MSE in CIELAB coordinates.Source: Figure 9 · p. 27

Related papers

Loading 2609.30633v1…