Source-linked AI summary
Phase Transition from Random Features to Fully Trainable Networks
Xin He, Huan Lao, Xiliang Lu, Shijun Zhang
TL;DR
Partially trainable networks bridge fixed random features and fully trainable networks, but how their approximation performance changes with the trainable proportion is the central question. The paper analyzes two-layer approximation bounds and examines training behavior, finding substantial gains at small trainable proportions and sharper transitions with greater initialization mismatch. In the spherical biased-ReLU example, about 4.1% trainable neurons capture 90% of the bound reduction from the fixed to fully trainable endpoint.
Problem
The paper asks how approximation performance changes as the trainable proportion varies between fixed random features and fully adaptive networks.
Source: §1 · p. 2Method
The paper analyzes two-layer P-trainable network approximation bounds using separate fixed and adjustable sampling groups, and examines a structured three-layer example and numerical training behavior.
Source: §1 · pp. 2–3, §1.1 · p. 4Results
For d=5 and τ=10, about 4.1% trainable neurons capture 90% of the approximation-bound reduction from P=0 to P=1, while experiments show similar small-P transitions.
Source: §1 · pp. 2–3Takeaways & Limitations
A small trainable group can substantially reduce approximation bounds when its mismatch is much lower than the fixed group's, with greater mismatch sharpening the transition.
Source: §1 · p. 2, §7 · p. 30The approximation bounds use a distributional construction that does not characterize the parameters produced by gradient-based training.
Source: §7 · p. 31
Abstract
from arXiv · showhide
Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $\alpha>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((\kappa_P\log(m/\delta)/m)^\alpha)$ for two-layer networks of width $m$ and trainable proportion $P$, where $\kappa_P=(P/\kappa_1^*+(1-P)/\kappa_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-\delta$. The constants $\kappa_0^*$ and $\kappa_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $\alpha=(d+2)/(d-1)$ and $\kappa_0^*=4\sqrt{2}\tau(d+1)/d$ for initialization scale $\tau\ge1$, and $\kappa_1^*=1$ under ridge leverage sampling. For $d=5$ and $\tau=10$, a trainable proportion of approximately $4.1\%$ captures $90\%$ of the reduction in the approximation bound from $P=0$ to $P=1$. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.
1. Introduction
The paper studies how approximation performance changes as hidden neurons shift from randomly fixed to trainable, connecting random-feature and fully trainable networks. Its analysis and experiments find substantial gains at small trainable proportions followed by diminishing returns, with sharper transitions under greater initialization mismatch.
1. Introduction
- A small trainable group can offset initialization mismatch, producing substantial gains at small P followed by diminishing returns; greater mismatch sharpens the transition.The analysis and experiments examine hidden parameters divided between trainable and randomly fixed groups.Source: §1 · p. 2
- For d=5 and τ=10, about 4.1% trainable neurons capture 90% of the approximation-bound reduction from P=0 to P=1.The spherical biased-ReLU analysis supplies explicit rates for initialization dependence.Source: §1 · p. 2, §1.1 · p. 4
1.1. Main Results
- A structured three-layer example shows that a few trainable first-layer neurons can use fixed downstream neurons through nonlinear composition.At selected trainable-neuron counts, its explicit error upper bound is smaller than that of a two-layer counterpart.Source: §1 · p. 2, §1.1 · p. 4
- Across target functions and architectures, numerical experiments show similar small-P transitions, including width and depth comparisons and networks up to 15 layers.Most tested configurations recover a substantial fraction of endpoint loss reduction at small P, followed by diminishing gains.Source: §1 · p. 3, §1.1 · p. 4
- For two-layer networks, the analysis gives approximation bounds over P∈[0,1]; under polynomial spectral decay and leverage compatibility, the rate depends on the mismatch of both neuron groups.The trainable group's sampling distribution is adjustable, while the fixed group uses its initialization distribution.Source: §1.1 · pp. 3–4
1.2. Related Work
- Unlike subnetwork search or parameter-selection studies, this setting retains all neurons and studies approximation capacity as the trainable proportion varies.The prescribed proportion of hidden neurons is trainable; the remaining hidden parameters stay fixed after initialization.Source: §1.2 · p. 6
2. Preliminaries and Problem Setting
The setup defines a network class with neuron-wise trainability, an approximation task, and an RKHS target space. For its analysis, hidden parameters are sampled from separate fixed and adjustable distributions, and approximation is bounded over those constructions.
2.1. Model Space: 𝑷-Trainable Neural Networks
- A P-trainable network assigns a proportion P of neurons in each hidden layer to the trainable group and the rest to a randomly fixed group, with widths held fixed across P.At P=0 the hidden parameters are fixed and at P=1 they are all trainable; output coefficients remain trainable.Source: §2.1 · pp. 7–8
2.2. Approximation Task: Two Complementary Views
- The approximation task has finite-sample and population views: the former evaluates prescribed input points, while the latter measures error in L2(dμ).With trainable hidden parameters, joint optimization of hidden parameters and output coefficients is generally nonlinear and non-convex.Source: §2.2 · pp. 8–9
2.3. Target Function Space: The Reproducing Kernel Hilbert Space
- The target belongs to the RKHS induced by the feature kernel, with the associated feature and integral operators providing the population-analysis framework.The target RKHS is viewed as a subspace of L2(X,dμ).Source: §2.3 · pp. 9–10
2.4. Distributional Approximation Strategy
- For approximation analysis, fixed neurons are sampled from prescribed ρ0 and trainable parameters are constructed using adjustable ρ1, with fixed group sizes and trainable proportion P.The two-group construction differs from i.i.d. sampling from the mixture because its group sizes are fixed; each construction lies in the P-trainable network class.Source: §2.4 · p. 11
- The analysis chooses output coefficients by population ridge regression after sampling hidden features.This isolates a tractable approximation construction; the distributional construction is for analysis and does not require training to optimize over distributions.Source: §2.4 · pp. 11–12
3. General Approximation Results
The section develops uniform approximation guarantees for partially trainable networks, then shows how spectral decay and sampling mismatch govern the bound’s dependence on trainable proportion. When trainable features have lower mismatch, a small trainable proportion can capture much of the available bound reduction.
3.1. Approximation Bounds for General Feature Maps
- Reweighting the sampled features preserves the kernel and its integral operator, allowing the analysis to retain the target RKHS while changing the sampling scheme.The construction then chooses output coefficients by population ridge regression.Source: §3.1 · p. 12
- A sampled hidden-parameter realization supports uniform approximation over the target RKHS unit ball, with output coefficients chosen separately for each target.The reweighted representation constrains the squared coefficient norm to at most 4/m.Source: §3 · p. 12, §3.1 · p. 13
3.2. Approximation Rates under Polynomial Spectral Decay
- The mismatch factor compares the largest ridge-leverage score with its sampling average, so insufficient sampling mass in some parameter regions increases mismatch.Figure 4 schematically contrasts a density that follows the ridge-leverage density with one that undersamples near a peak.Source: §3.2 · p. 14, Figure 4 · p. 14
- O((m/log m)^-α) is the width-dependent bound rate for fixed trainable proportion, confidence, and mismatch constants; the trainable proportion enters through a non-increasing mismatch-dependent prefactor.Polynomial spectral decay and leverage compatibility yield the explicit rate; improving the trainable group’s mismatch relative to the fixed group can substantially reduce the bound.Source: §3.2 · pp. 15–16
3.3. Small-𝑷Phase Transition
- The normalized gain increases with trainable proportion but has diminishing marginal improvement; when the mismatch ratio is large, the threshold for a fixed gain fraction decreases as O(r^-1).At r=100, the threshold for 90% normalized gain is approximately 2.2%.Source: §3.3 · pp. 16–17
4. Analysis of Spherical Biased-ReLU Features
The section verifies the spectral-decay and sampling-compatibility conditions for spherical biased-ReLU features, then uses them to quantify how initialization mismatch affects approximation gains. Increasing the mismatch sharpens the predicted transition, with a small trainable proportion capturing most of the gain in the illustrated settings.
4. Analysis of Spherical Biased-ReLU Features
- The spherical setting verifies the spectral-decay and sampling-compatibility assumptions needed to specialize the general approximation rates.The spectral exponent is α=(d+2)/(d−1), and the admissible fixed-group mismatch constant increases with τ.Source: §4 · p. 18, §4.2 · p. 21
4.1. The Target RKHS and Kernel Spectrum
- Scaling fixed-neuron initialization by τ leaves the target RKHS unchanged, shifts parameters toward b=±1, and yields a mismatch that grows linearly with τ.The trainable group uses optimal ridge leverage sampling, with κ*_1=1.Source: §4.1 · pp. 18–19, §4.2 · p. 21
- The target space is induced by a rotation-invariant ReLU kernel whose integral operator is compact and positive semidefinite, with spherical harmonics as eigenfunctions.The kernel spectrum is analyzed through the spherical harmonic decomposition.Source: §4.1 · pp. 18–20
4.2. Initialization Mismatch and Approximation Rates
- At d=20 (α=22/19), the predicted 90%-gain threshold falls from approximately 0.563 at τ=1 to 0.100 at τ=10.The figure plots normalized gain against trainable proportion P, with guides at 90% gain and G(1)=1.Source: §4.2 · p. 21, Figure 7 · p. 21
- For d=5 and τ=10, about 4.1% trainable neurons capture 90% of the reduction in the approximation bound from P=0 to P=1.The observed gain curve for τ=10 rises more rapidly at small P than for τ=1, qualitatively matching the bound's prediction.Source: §4.2 · p. 21
5. A Structured Three-Layer Illustration
A structured three-layer network illustrates how a small trainable first-layer group can use fixed downstream neurons through nonlinear composition. Its approximation bounds show rapid gains at small trainable counts, while the example does not establish a general deep-network theory.
5. A Structured Three-Layer Illustration
- The comparison uses 220 hidden neurons and four trainable neurons, placing first-layer pairs before fixed blocks in one architecture and reallocating the second-layer budget to one layer in the other.The first-layer neurons form 10 pairs, each feeding a separate block of 20 fixed neurons with tied incoming weights.Source: Figure 8 · p. 23, §5 · p. 23
- With two trainable neurons, both constructions attain approximately 80% of normalized bound reduction; with four, their gains are 0.9997 and 0.9877, respectively.With four trainable neurons, the three-layer error upper bound is 2.06% of its two-layer counterpart under matched neuron budgets.Source: §5 · p. 24, Table 1 · p. 24
- The three-layer construction uses four trainable first-layer neurons to adapt 40 downstream basis functions through fixed second-layer blocks and nonlinear composition.Each trainable neuron generates a feature that feeds a block of 20 fixed neurons; the two-layer comparison uses trainable neurons directly as interpolation basis functions.Source: Figure 8 · p. 23, §5 · p. 24
- A general theory for phase transitions and approximation advantages of nonlinear composition in multilayer trainable networks remains undeveloped.Shared upstream parameters couple downstream features through nonlinear composition, complicating a direct deep-layer extension of the probabilistic kernel argument.Source: §5 · pp. 22, 24
6. Numerical Evidence of Phase Transitions
Across varied targets and architectures, training losses show a small-P phase transition: substantial early reductions give way to diminishing gains. Experiments also test other activations and the contribution of randomly fixed neurons.
6.1. Experimental Setting
- Experiments evaluate neuron-wise trainability using P-trainable networks across three targets with different oscillation scales and input dimensions, widths 32, 64, 128, and 256, varied depths, and 12,000 steps over 21 repetitions.Source: §6.1 · pp. 25–26
6.2. Experimental Results
- Most configurations exhibit substantial loss reductions at small P, followed by diminishing gains as normalized gain curves approach a plateau; this pattern is particularly clear for f*1 and deeper networks on f*2 and f*3.Source: §6.2 · p. 29
- For high-frequency f*2, two- and three-layer networks retain losses above 10^-1 across tested P, so their normalized-gain curves are omitted.The three-layer loss fluctuates with P; neither architecture fits the target well under these settings.Source: Figure 11 · p. 28, §6.2 · p. 29
6.3. Results with Other Activation Functions
- GELU and tanh recover over 95% of endpoint loss reduction at P=1/8 for all three targets at depth 5 and width 64.This is consistent with the small-P transition observed with ReLU.Source: §6.3 · p. 29, Table 3 · p. 30
6.4. Expressive Role of Randomly Fixed Neurons
- With 32 trainable neurons per hidden layer, increasing randomly fixed neurons from width 32 to 256 reduces loss by approximately 94%.The results support an expressive role for randomly fixed neurons, although total trainable parameter counts vary with width.Source: §6.4 · pp. 29–30, Table 4 · p. 30
7. Discussion and Conclusions
The paper connects approximation bounds, structured network examples, and numerical phase-transition patterns, while noting limits to the theory's reach. Its bounds rely on a parameter-distribution construction that does not characterize gradient-trained parameters, and a general theory for deeper networks remains undeveloped.
7. Discussion and Conclusions
- Numerical experiments show phase-transition patterns across targets and architectures and support an expressive role for randomly fixed neurons.The two-layer bounds quantify how sampling mismatch affects approximation; a structured three-layer example uses nonlinear composition to obtain a smaller explicit bound.Source: §7 · p. 30
- The approximation bounds rely on a distributional construction that does not characterize parameters produced by gradient-based training.A general phase-transition theory for deeper trainable networks also remains undeveloped, in part because upstream updates change inputs to later fixed neurons.Source: §7 · p. 31
Overview of the Appendices
The appendices collect notation, proofs, derivations, and supplementary experiments.
Overview of the Appendices
- Appendix A summarizes notation, Appendices B–D provide proofs and error-bound derivations, and Appendix E contains additional experiments.Appendix B covers general approximation bounds and polynomial rates; Appendix C treats spherical ReLU spectral and mismatch bounds; Appendix D covers structured examples.Source: Overview of the Appendices · p. 31
A. Additional Technical Notes
Table 5 collects the paper's main notation.
A. Additional Technical Notes
- Table 5 summarizes the main notations used in the paper.Source: §A.1 · p. 31
A.2. Clarifications of the Assumptions
The assumptions supply operator regularity, polynomial spectral decay, and sampling compatibility needed for the approximation analysis. The proof outline combines operator concentration with ridge construction, then uses spectral decay and compatibility to obtain polynomial rates.
A.2. Clarifications of the Assumptions
- Assumption 2.2 makes the feature operator Hilbert–Schmidt, so the integral operator is self-adjoint, positive, trace-class, and compact.These properties justify spectral decomposition and make the effective dimension finite for every positive regularization parameter.Source: §A.2 · p. 31
- Leverage compatibility requires each sampling density to have a constant-factor bound relative to the optimal ridge leverage density, strengthening a maximal-score bound.Under polynomial spectral decay, the condition implies the related bound with C*=1/α for sufficiently small λ.Source: §A.2 · pp. 32–33
- Assumption 3.5 imposes two-sided eigenvalue decay, with the upper bound controlling the approximation rate and the lower bound identifying the actual polynomial scale.The assumption yields d(λ) ≍ λ^-1/α; trace-class alone does not guarantee this decay.Source: §A.2 · p. 33
Outline
- The proof outline uses operator Bernstein concentration for independently sampled feature groups, then combines ridge construction, mixture leverage, compatibility, and spectral decay.The variance-proxy Bernstein inequality applies to independent, centered, self-adjoint Hilbert–Schmidt random operators under operator-norm and variance bounds.Source: Outline · p. 33, §B.1 · p. 33
B.2. Proofs of the Main Results in Section 3
The proof extends the ridge-based approximation analysis to two independently sampled feature groups and establishes uniform control over the RKHS unit ball. It then combines concentration, leverage compatibility, and polynomial spectral decay to derive rates, alongside spherical harmonic calculations for the spherical setting.
B.2. Proofs of the Main Results in Section 3
- The proof adapts the general approximation strategy to two independent feature groups sampled separately from ρ1 and ρ0, using a variance-proxy Bernstein inequality.The source strategy assumes one i.i.d. feature distribution, so the proof modifies it for the two-group setting.Source: §B.2 · p. 37
- The ridge solution reduces approximation error and coefficient-norm control to a common quadratic form, which operator concentration bounds by the target’s squared RKHS norm.Because the concentration event is target-independent, the resulting bounds hold uniformly over the RKHS unit ball.Source: §B.2 · p. 37
- The effective dimension has polynomial spectral decay, d(λ) ≍ λ^-1/α; its matching lower bound follows from d(λ) ≥ J_{λ/2} ≍ λ^-1/α.Source: §B.2 · p. 41
C.2. Proofs of Main Results in Section 4
The spherical proof derives the ReLU kernel’s spectrum and initialization mismatch, then applies the general approximation theorem with leverage sampling. These ingredients establish explicit spherical approximation rates.
C.2. Proofs of Main Results in Section 4
- For the spherical biased-ReLU kernel, degree-wise eigenvalue decay combined with harmonic multiplicities yields the ordered spectrum’s polynomial decay.The proof overview attributes the eigenfunctions to rotation invariance and the Funk–Hecke formula, and obtains the ordered decay by counting multiplicities.Source: §C.2 · pp. 48–50
- Uniform spherical initialization has mismatch bounded asymptotically by 2(d+1)/d plus ε for sufficiently small λ.The bound applies for every ε>0, with the sufficiently small λ range depending on ε.Source: §C.2 · p. 50
- The scaled spherical initialization result follows by applying the uniform-initialization mismatch bound with ε/τ.The proof states the resulting bound holds for all sufficiently small positive λ.Source: §C.2 · pp. 52–53
- The spherical rate proof uses normalized ridge leverage sampling for trainable neurons and the scaled-initialization mismatch bound for fixed neurons.When its regularization choice is within the specified small-λ range, Theorem 3.6 yields the rate; the same rate holds in both cases considered.Source: §C.2 · pp. 53–54
D. Error Bounds for the Structured Three-Layer Example 5.1
The structured example uses trainable neurons to build coordinate-wise interpolation, including a three-layer construction that reuses fixed downstream blocks. The initialization-scale experiment reports a similar dependence in numerical training gains and theoretical approximation bounds, with quantitative differences expected between them.
D. Error Bounds for the Structured Three-Layer Example 5.1
- Four trainable first-layer neurons form |x2| and |x3|, letting fixed second-layer blocks implement piecewise-linear interpolation for both target coordinates.The construction uses random biases as interpolation nodes and sets output coefficients of other blocks to zero.Source: D. Error Bounds for the Structured… · p. 55
- With 20 trainable first-layer neurons, five pairs feed each coordinate into five fixed blocks, pooling 100 random biases per coordinate for interpolation.All second-layer parameters remain fixed.Source: D. Error Bounds for the Structured… · p. 58
- The two-layer construction’s expected squared L2 error is at most 1/[9(m_free/2 + 1)^4] for m_free trainable neurons.The construction uses symmetric ReLU pairs to interpolate the squared coordinate function; for two neurons, the stated upper bound is 17/729.Source: D. Error Bounds for the Structured… · p. 59
- For m_free=2 and 4, the three-layer gains are approximately 0.7998 and 0.9997, while the two-layer gains are approximately 0.7902 and 0.9877.The source reports these values as G^(3)(2), G^(3)(4), G^(2)(2), and G^(2)(4), respectively.Source: D. Error Bounds for the Structured… · p. 59
E.1. Effect of the Initialization Scale
- The numerical threshold P0.1 decreases overall as τ increases, and τ=10 gains rise faster at small P than τ=1 gains.The numerical curves concern training one fixed target under finite sampling and a fixed budget, whereas theoretical curves bound approximation over the RKHS unit ball; quantitative differences are expected.Source: Figure 1 · p. 61, §E.1 · p. 61
E.2. Additional Experiments with Other Activation Functions
With GELU and tanh, the additional width and depth experiments show small-P phase-transition patterns similar to those observed with ReLU; small loss differences do not change the overall pattern.
E.2. Additional Experiments with Other Activation Functions
- With GELU and tanh, width and depth experiments show small-P phase-transition patterns similar to those observed with ReLU.The experiments use the same targets, architectures, and training settings as Section 6.1.Source: §E.2 · p. 61
- For two-layer GELU, partial training sometimes gives slightly lower loss than full training, but the differences are only about 10^-5 and do not alter the overall pattern.Normalized gains are slightly above 1, so the gain curve is omitted.Source: §E.2 · p. 61
- Figures 13–18 present training results for f* in the additional experiments.The section reports patterns similar to those observed with ReLU.Source: Figure 13 · p. 62, Figure 14 · p. 62, Figure 15 · p. 63