PaperlayerView exact revision on arXiv

Source-linked AI summary

The role of parameter Jacobians in the stability of network outputs

Halyun Jeong, Palle E. T. Jorgensen, Hyun-Kyoung Kwon, Myung-Sin Song, James Tian

arXiv:2608.27748v1math.FAstat.ML

TL;DR

The paper addresses how restricting Jacobian parameter directions changes frozen NTK output dynamics and develops a semigroup-based perturbation framework. It proves explicit task-local comparisons for finite-time and averaged evolutions, while extending the analysis to spectral-distribution, unbounded, and time-dependent settings. The results provide a bounded frozen-model theory, with nonlinear kernel drift and systematic unbounded extensions remaining outside scope.

  • Problem

    The paper asks how modifying the tangent feature map by restricting parameter directions changes frozen NTK output dynamics on prescribed task spaces.

  • Method

    It represents linearized training with Jacobian Gram operators and compares full and restricted contraction semigroups using task-local perturbation theory.

  • Results

    The paper proves explicit finite-time and Cesàro comparison estimates, including task-space refinements, low-frequency spectral conditions, and piecewise-frozen nonautonomous extensions.

  • Takeaways & Limitations

    The framework quantifies how pruning or restricting tangent directions affects frozen linearized output evolution on task-relevant spaces.

  • Takeaways & Limitations

    The averaged results concern the frozen model, while empirical kernel drift and systematic unbounded analogues are left outside the developed theory.

Abstract

from arXiv · show

In the framework of network dynamics, learning models, and neural tangent kernels (NTK), we show that the corresponding linearized dynamics leads naturally to a semigroup formulation. More precisely, in our analysis of input/output models, the time-dynamics is presented via special semigroups of linear operators on Hilbert spaces, together with an associated class of semigroup perturbations. In this context, we then present new and explicit a priori perturbation-bound results: for the fixed-kernel linearization constructions arising in the NTK setting, we prove norm-bounds on the corresponding semigroup perturbations, in the form of explicit finite-time perturbation estimates. We further present refinements on prescribed task spaces, Cesàro-averaged (ergodic) comparisons estimates, and versions in which the lower spectral edge assumption is replaced by a spectral-distribution condition. We also extend the comparison to nonautonomous NTK evolutions through piecewise-frozen approximations, record a corresponding discrete Euler specialization, and offer worked examples in order to illustrate our perturbation-bound estimates.

1. Introduction

The paper develops task-local operator-theoretic perturbation bounds for Jacobian-induced output dynamics, focusing on how restricting parameter directions affects network outputs. It covers finite-time, averaged, spectral-distribution, unbounded, and time-dependent extensions.

  • Motivation: The paper studies how restricting, freezing, or pruning parameter directions perturbs network-output stability and approximation error through bounded linearizations on Hilbert spaces.The analysis quantifies propagation from parameter-side perturbations to output-side dynamics.
  • Scope boundary: For exact hard truncation, discarded tangent directions may fail to remain uniformly small relative to the full tangent norm on the entire cyclic task space.The mixed condition accommodates relative loss together with additive task-space error.
  • Fixed-kernel NTK setting: In the fixed-kernel NTK model, pruning replaces T with TP and changes the output generator from G = TT∗ to GP = TPT∗.The resulting comparison concerns the full and restricted semigroups on a prescribed task space.
  • Finite-time estimates: The paper derives explicit finite-time bounds under relative task capture and a complementary finite-dimensional comparison separating spectral attenuation from off-diagonal mixing.The relative condition controls discarded tangent energy on task-coupled directions; the finite-dimensional refinement does not require it.
  • Averaged estimates: Cesàro comparisons approximate the original average by the pruned average, identify persistent stationary task components, and use either a positive lower spectral edge or low-frequency spectral-mass bounds.The averaged estimates measure effects distinct from fixed-horizon deviations.
  • Extensions: The framework extends to suitable unbounded generators and time-dependent Jacobians through piecewise-frozen approximations, with discrete Euler and worked-example specializations.A systematic unbounded perturbation theory remains outside the paper’s scope.

2. The fixed-kernel linearization and the NTK picture

The paper formulates frozen NTK training as residual evolution under Jacobian Gram-operator semigroups. It compares full and parameter-restricted evolutions on task-relevant cyclic subspaces using task-capture and finite-dimensional perturbation analyses.

  • Fixed-kernel model: At a reference parameter, the fixed Jacobian T induces the positive output-side Gram operator G = TT∗ and residual semigroup e−tG.This is the frozen or lazy-training linearization used throughout the paper.
  • Parameter restriction: Projecting onto active parameter directions replaces T by TP, yielding GP = TPT∗ and restricted residual evolution rP(t) = e−tGP rP(0).The projection freezes the remaining parameter directions.
  • Comparison problem: The central comparison is between full and restricted residual evolutions e−tTT∗ and e−tTPT∗, rather than the full nonlinear training process.The Jacobian is held fixed at the reference point.
  • Task space: The task space is enlarged to the closed G-cyclic subspace generated by M, the smallest closed G-invariant space containing the prescribed output directions.This space captures output directions reached by the autonomous unperturbed flow.
  • Finite-time assumptions: Relative task capture requires discarded tangent components to be at most an ϵ-fraction of the full tangent norm on task-coupled directions.Under this hypothesis, restricted residuals remain close to full residuals over [0, τ].
  • Finite-dimensional refinement: A complementary finite-dimensional comparison uses common-task-space invariance to separate attenuation within spectral modes from mixing between distinct modes.This comparison does not assume the task-capture condition.
  • Nonautonomous boundary: For time-dependent generators, the autonomous cyclic-space horizon independence may fail because later evolution can couple tasks to previously unreached directions.This motivates a finite-horizon task-space construction.

3. Finite-time perturbation estimates

The paper formulates pruning as a perturbation of fixed-kernel semigroup dynamics and derives finite-time output-stability bounds on task-relevant subspaces. It also develops soft-pruning and finite-dimensional spectral refinements.

  • Finite-time perturbation estimates: Theorem 3.1 uses Duhamel’s formula to obtain an a priori norm estimate comparing the original and pruned semigroups.The comparison is evaluated on the cyclic task space generated by M over a finite horizon.
  • Finite-time perturbation estimates: The operators are G = TT ∗, GP = TPT ∗, and QP = G −GP = T(I −P)T ∗, representing the unpruned generator, pruned generator, and perturbation.P retains active parameter directions and removes directions in kerP.
  • Finite-time perturbation estimates: Under ∥(I −P)T ∗u∥H ≤ϵ∥T ∗u∥H on the cyclic task space, the semigroup comparison is controlled using contraction properties and pointwise orbit estimates.The hypothesis may be imposed on a larger closed G-invariant subspace, at the cost of potentially strengthening capture requirements and increasing the spectral constant.
  • Finite-time perturbation estimates: Soft pruning replaces the projection P with a positive contraction A, allowing parameter directions to be attenuated rather than only retained or removed.The corresponding operators are GA = TAT ∗, QA = T(I −A)T ∗, and SA(t) = e−tGA.
  • Finite-time perturbation estimates: In finite dimensions, finitely many semigroup snapshots in [0, σ] span the cyclic task space, with q snapshots associated with the distinct eigenvalues of G.The spanning argument uses an invertible Vandermonde matrix formed from the distinct exponential factors.
  • Finite-time perturbation estimates: The perturbation theorem only requires control on spectral directions coupled to M, rather than necessarily on the whole output space K.When M = K, the cyclic task space is K and the result becomes global.

4. Refinement on the task space

The task-space refinement decomposes pruning error into diagonal attenuation and off-diagonal spectral mixing on an invariant cyclic subspace. This yields bounds that do not require the relative task-capture condition used earlier.

  • Refinement on the task space: The finite-dimensional refinement assumes GP N ⊆N and splits GP|N = D + R into diagonal and off-diagonal parts relative to G’s spectral decomposition.D preserves spectral blocks, while R mixes distinct G-eigenspaces.
  • Refinement on the task space: Unlike Theorem 3.1, this comparison does not assume the relative task-capture condition (3.1); it quantifies diagonal attenuation and off-diagonal mixing through δ and ρ.The cyclic task space N is independent of the comparison horizon.
  • Refinement on the task space: The resulting estimate is ∥(SP (t) −S (t)) |M∥≤t (δλN + ρ), t ≥0.The corresponding bound holds uniformly on [0, τ] after evaluating the right-hand side at τ.
  • Refinement on the task space: The nonlinear-in-time form separates the effects as 1 −e−tδλN for diagonal attenuation and tρ for off-diagonal mixing.If R = 0, the comparison reduces to the modewise bound 1 −e−tδλN.
  • Refinement on the task space: Blockwise mixing is small when each task-coupled G-mode has small total coupling to other task-coupled modes in both row and column directions.A value δ < 1 supplies additional quantitative information rather than an existence condition.
  • Refinement on the task space: If row and column mixing are each bounded by η, the estimate becomes ∥(SP (t) −S (t)) |M∥≤1 −e−tδλN + tη.This gives a task-space comparison with a single additive mixing parameter.

5. Ces`aro averages and ergodic comparison on the task space

The ergodic section studies Cesàro averages of the unpruned and pruned semigroups, identifying their stationary component and bounding averaged tangent leakage. It replaces uniform relative control with a mixed condition when hard truncation violates the stronger hypothesis.

  • Ces`aro averages and ergodic comparison on the task space: Cesàro averages are used to compare CP,L|M with CL|M and identify stationary components that may remain in their difference.The relevant limits are derived directly by spectral calculus because G is bounded, self-adjoint, and positive.
  • Ces`aro averages and ergodic comparison on the task space: For exact hard truncation, the relative condition (3.1) may fail because discarded tangent energy need not be uniformly small relative to ∥T ∗u∥H.The paper therefore introduces a weaker mixed task-tail condition with relative constant α and additive constant β.
  • Ces`aro averages and ergodic comparison on the task space: The additive term β can be absorbed only when a positive lower task-energy bound is available.Without uniform ambient task-norm control, its usefulness is instead tied to decay of averaged task energy.
  • Ces`aro averages and ergodic comparison on the task space: The frozen unpruned average converges strongly to Pker G = Pker T ∗ as L →∞.This follows from the spectral behavior of the averaging function, which tends to zero on positive spectral values and equals one at zero.
  • Ces`aro averages and ergodic comparison on the task space: The ergodic analysis also covers operators U satisfying UPker G = 0, extending the stationary-component comparison beyond the identity output.The kernel relation ker G = ker T ∗ underpins the identification of the limiting projection.
  • Ces`aro averages and ergodic comparison on the task space: The mixed condition yields an averaged leakage bound along the unpruned orbit without requiring N to be invariant under GP.A separate comparison of CP,L and CL does require the invariance hypothesis.

0 S(t)v dt, Jensen’s inequality gives

The Cesàro average of the pruned–original semigroup difference converges to a stationary projection, and vanishes exactly when pruning creates no task-visible stationary directions.

  • The kernel inclusion ker(G|N) ⊆ ker(GP|N) makes the unpruned stationary subspace invariant under the pruned operator.Consequently, the complementary nonstationary space N+ is also invariant under GP|N.
  • The averaged semigroup difference converges in operator norm to the stationary projection term PE|M.This identifies the exact long-horizon contribution that remains after averaging.
  • The Cesàro average tends to zero exactly when pruning creates no new stationary directions visible from M.The limiting difference is governed by the stationary subspaces of G|N and GP|N.
  • When stationary projections agree on M, the averaged pruned dynamics converge in operator norm to the averaged original dynamics.The convergence rate is determined by the low-frequency spectral behavior under the relevant hypotheses.

6. Ces`aro estimates without a lower spectral edge

The paper replaces a positive lower spectral edge with task-dependent control of low-frequency spectral mass, obtaining explicit decay and averaged-dynamics comparison bounds.

  • Low-frequency spectral-distribution conditions replace the positive lower spectral edge used in earlier Cesàro decay estimates.The weaker conditions control nonzero spectral mass near the origin on the task space.
  • The quantity eF B v (r) measures how much of v lies in the nonzero spectral window (0, r].This excludes the stationary zero spectral atom from the low-frequency measure.
  • Under a power bound on low-frequency spectral mass, Theorem 6.3 gives an explicit inverse-horizon estimate for averaged spectral energy.The bound holds for L ≥∥B∥−1 and depends on γ, ∥B∥, and Cv.
  • The mixed task-tail condition controls discarded components of the Cesàro-averaged unpruned tangent vector.Taskwise spectral control implies asymptotic concentration in the retained parameter space ran P.
  • When stationary projections agree on M, the averaged pruned dynamics approximate the original dynamics in operator norm at a rate set by the origin exponents.The general comparison first subtracts the stationary projection difference.

7. A finite-time estimate for unbounded generators

The finite-time Duhamel–energy argument extends to self-adjoint, possibly unbounded generators when variation of constants and an orbitwise relative square-root bound are available.

  • The extension treats unbounded task generators arising naturally from derivative-weighted losses.Derivative operators can have norms scaling as h−1 or h−2 as pixel spacing h decreases, unless the kernel smooths high frequencies.
  • A systematic theory for constructing perturbed unbounded generators, verifying domain conditions, and proving averaged or nonautonomous analogues is left for future work.The present extension is restricted to finite-time comparisons.
  • The perturbation is controlled along task trajectories by the relative estimate ∥WS(s)v∥K ≤η∥A1/2S(s)v∥K.The theorem also assumes self-adjoint nonnegative generators and a valid variation-of-constants formula.
  • Semigroup regularization places S(s)v in D(A1/2) for every s > 0, allowing the orbitwise bound to enter the Duhamel estimate.Contractivity of the perturbed semigroup then yields the finite-time comparison bound.
  • The bounded-generator specialization recovers the two forms of the earlier finite-time estimate.The additional unbounded-case ingredient is regularization by S(s).

8. Nonautonomous NTK evolution and piecewise-frozen approximation

For time-varying NTK dynamics, the paper constructs contractive evolution families and approximates them by products of semigroups obtained by freezing the Jacobian on partition subintervals.

  • At finite width, the empirical NTK may evolve substantially, motivating a nonautonomous treatment beyond the fixed-kernel approximation.The paper therefore allows the Jacobian and generator to vary with time.
  • The nonautonomous residual equation r′(t) = −G(t)r(t) generates a unique contractive evolution family U(t,s).Positivity of G(t) implies contractivity.
  • The relevant task space is a finite-horizon orbit hull because later evolution may activate directions absent at earlier times.This horizon dependence contrasts with the autonomous cyclic task space.
  • The piecewise-frozen propagator is a product of local factors e−∆tkGk over a partition of [0,τ].Reference points ξk determine the frozen generators on each subinterval.
  • Under Hölder continuity of G(t), midpoint freezing yields a partition-dependent approximation rate of order |Π|^α.Theorem 8.2 supplies the telescoping-product comparison underlying this estimate.
  • Intervalwise parameter projections yield pruned piecewise-frozen bounds based on local tangent capture and propagated task spaces.In finite dimensions, attenuation and off-diagonal mixing can also be decomposed and summed interval by interval.

9. A discrete Euler specialization

The paper specializes its nonautonomous NTK comparison to discrete Euler products, deriving parallel perturbation estimates and identifying projection-based algorithms as special cases.

  • Discrete Euler specialization: Replacing each local semigroup factor e^−Δt_kG_k with I −η_kG_k yields a discrete-time perturbation estimate for pruned products.The discrete model highlights that, for non-small step sizes, gradient descent is an iteration rather than a continuous flow.
  • Discrete Euler specialization: The output-side products are formed by composing Euler factors I −η_kG_k and their pruned counterparts over successive time steps.The construction also propagates discrete task spaces, with each factor assumed to be contractive.
  • Discrete Euler specialization: Theorem 9.1 assumes per-step task-local errors ε_k in [0,1) and compares the full and pruned Euler products through contraction and telescoping arguments.The proof uses factorization, Cauchy–Schwarz, contraction of the factors, and a telescoping identity.
  • Projection specializations: Euler products include output-side projection algorithms of Kaczmarz type as a special case.When local kernel operators are rank-one or low-rank projections, the dynamics become kernel Kaczmarz or block-Kaczmarz products.
  • Projection specializations: For projection operators, the Euler step becomes a classical projection step, while 0 < η_kλ_k ≤ 2 gives a relaxed Kaczmarz step.The resulting output-side product is a special projection example related to operator and block Kaczmarz methods.

Appendix A. A worked example

A finite-dimensional example computes the perturbation constants for a non-diagonal Jacobian and a task space that occupies only part of the spectrum, illustrating complementary attenuation and mixing bounds.

  • Worked example: The example uses a non-diagonal Jacobian and a cyclic task space occupying only part of the spectrum of G.It is designed to illustrate both the discarded-kernel estimate and the attenuation/mixing refinement.
  • Worked example: The task space is M = span {e_2, e_3}, is G-invariant, and generates the minimal invariant subspace used in the comparison.The parameter projection removes one of four parameter directions.
  • Worked example: The task space sees eigenvalues 4 and 1, whereas the global norm is governed by an inactive mode with eigenvalue 36.The finite-time estimate instead depends on the discarded-kernel norm ∥Q_P∥ = 5/4.
  • Attenuation and mixing: The example computes the diagonal contribution from the e_2- and e_3-modes and identifies t/2 as the off-diagonal mixing term.Thus the abstract decomposition from Section 4 is realized explicitly in the example.
  • Bound comparison: The appendix compares the actual restricted-versus-full operator norm with the bounds from Theorems 3.1 and 4.1 at t ∈ {1/4, 1/2, 1}.The constants ε, δ, and ρ are reported as optimal for the respective hypotheses, while the discarded-kernel norm is exact.
  • Attenuation and mixing: The refined estimate separates diagonal attenuation, measured by δ, from off-diagonal mixing, measured by ρ, without assuming the relative task-capture condition.This complements the Section 3 estimate, which uses task-local tangent capture.
Loading 2608.27748v1…