PaperlayerView exact revision on arXiv

Source-linked AI summary

A Commutator Framework for Selective Spectral Alignment in Deep Neural Networks

Kaj Nyström

arXiv:2608.22910v1stat.MLcs.LG

TL;DR

It asks how spectral compatibility is generated, transported, and cancelled in finite-width nonlinear networks, rather than assuming training universally eliminates incompatibility. The paper develops commutator identities and localization tools, finding selective, mechanism-dependent alignment rather than universal decay.

  • Problem

    The paper addresses how incompatibility among activation, covariance, sensitivity, and feature-side geometries is generated, transported, localized, and cancelled across layers.

    Source: §1 · p. 2

  • Method

    The paper develops a finite-width commutator framework with layerwise source decompositions, singular-value-weighted transport, buffered localized energies, and conditional decay estimates.

    Source: §1 · pp. 2–3

  • Results

    Feature-side alignment does not determine internal geometry, while finite-time observations show selective, mechanism-dependent alignment rather than universal commutator decay from risk minimization.

    Source: §1 · pp. 2, 5

  • Takeaways & Limitations

    Spectral alignment is a layer- and scale-dependent compatibility phenomenon governed by transport, interaction, cancellation, and possible damping.

    Source: §1 · pp. 3, 5

    Experimental conclusions are finite-time observations restricted to tested initializations, architectures, widths, datasets, and optimization horizon, without step-size or normalization comparisons.

    Source: §10.7 · p. 51

Abstract

from arXiv · show

We develop a finite-width geometric framework describing how learned feature geometries are organized, transported, and selectively aligned in deep neural networks. Incompatibility among weight-generated covariance, gates, and backward sensitivities is quantified through three families of commutators: between gates and covariance, between sensitivities and covariance, and between average gradient outer products (AGOPs) and neural feature matrices (NFMs). An exact layerwise identity decomposes the sensitivity-covariance commutator into four sources: downstream transport, adjacent-layer imbalance, pointwise sensitivity fluctuations, and nonlinear gate-covariance interactions. The AGOP-NFM commutator is a singular-value-weighted transport of the internal commutator, explaining why observed feature-side alignment alone does not determine the internal geometry from which it emerges. Buffered localized energies resolve mixing between separated covariance subspaces. We establish spectral-gap, projector-evolution, and stabilization estimates, and formulate conditional Lyapunov principles that yield decay under explicit geometric error-bound or intrinsic-damping assumptions. These criteria do not follow from gradient flow alone and clarify why risk reduction need not imply commutator collapse. Analytic examples and numerical experiments exhibit factorization of spectral and activation geometry, transient growth, and cancellation among nonzero sources. In tested finite-time regimes, cancellation dominated by a negative transport-imbalance interaction persists across depths, widths, and two regression benchmarks. Spectral alignment therefore appears as a layer- and scale-dependent compatibility phenomenon governed by transport, interaction, cancellation, and possible damping, rather than a universal consequence of training.

1. Introduction

This paper develops a finite-width commutator framework for explaining how spectral incompatibility is generated, transported, localized, and potentially reduced in nonlinear networks. It shows that feature-side alignment can conceal internal geometry, while decay requires mechanisms beyond gradient flow alone.

1. Introduction

  • The AGOP-NFM commutator is a singular-value-weighted transport of the internal sensitivity-covariance commutator, so feature alignment can suppress or hide internal mixing.Small internal incompatibility implies feature-side alignment subject to weight scale, but the converse can fail through low-variance filtering and covariance-null directions.Source: §1 · pp. 2–4
  • Buffered localized energies isolate direct coupling between separated covariance subspaces, remain invariant within those subspaces, and distinguish selective alignment from global commutativity.Buffers can exclude nearly degenerate boundary clusters from misalignment, while spectral-gap estimates relate localized defects to global commutator energies.Source: §1 · pp. 3–4, §1.1 · p. 5

1.1. Our principal contributions.

  • Stabilization of localized energies does not imply decay: convergence requires gap and regularity control, while genuine reduction needs compatibility, a geometric error bound, or intrinsic transverse damping.The framework separately treats smooth gate evolution and discontinuous ReLU switching, and identifies additional imbalance, fluctuation, operator-scale, and boundedness conditions for propagation.Source: §1 · p. 4, §1.1 · p. 5

1. Introduction

  • Finite-time experiments show risk reduction without universal commutator decay, width-dependent reductions in internal and feature defects but not gate defects, and persistent negative transport-imbalance cancellation.These observations, including function-preserving rescaling effects, support selective and mechanism-dependent alignment rather than universal decay driven by risk minimization.Source: §1 · pp. 4–5, §1.1 · p. 5

1.1. Our principal contributions.

  • An exact recursion decomposes internal incompatibility into downstream transport, adjacent-layer imbalance, pointwise sensitivity fluctuations, and nonlinear gate-covariance interactions that may reinforce or cancel.The gate commutator is an intrinsic nonlinear source, but it is neither the sole source nor a complete explanation of small total commutators.Source: §1 · p. 2, §1.1 · p. 5

2. Relation to existing work

This work weakens exact AGOP–NFM functional correspondence to eigenspace compatibility and studies how finite-width nonlinear dynamics generate, transport, or obstruct it. Its commutator framework complements prior feature-learning, balancedness, symmetry, and alignment approaches through exact layerwise identities, selective localized energies, and explicit finite-width scope.

Relation to feature-learning literature

  • Unlike prior AGOP, NFA, neural-collapse, tangent-feature, representation-alignment, and convergence-based accounts [4] [3] [2] [22], this framework derives the internal dynamics producing feature-side alignment.The cited literature motivates feature learning and observed geometric relations, whereas this work analyzes their finite-width dynamical origins.Source: §2 · pp. 6–7

Conceptual departure

  • The framework replaces exact AGOP–NFM spectral correspondence with weaker shared-subspace compatibility, separating the observable feature relation from the internal geometries that generate it.It tracks sensitivity, covariance, activation, imbalance, and fluctuation geometries without proposing a competing feature ansatz.Source: §2 · p. 6

Balancedness and symmetry

  • Deep-linear balancedness results [1] [11] [17] provide the principal exact comparison, while nonlinear networks generally retain only neuronwise norm-difference invariants [9] [12] [14].Accordingly, adjacent-layer matrix imbalance is treated as a source in the commutator recursion rather than imposed away, and rescaling can alter conserved imbalance values without changing the represented function.Source: §2 · p. 6

Dynamical comparison

  • The central dynamical contribution is an exact finite-width layerwise identity decomposing the internal sensitivity–covariance commutator into transport, imbalance, fluctuation, and gate–covariance sources.These identities require neither critical points nor balanced initialization, thereby allowing transient growth and cancellation among nonzero sources.Source: §2 · pp. 6–7

Selective alignment and scope

  • Buffered localized energies distinguish mixing across separated covariance subspaces from mixing within nearly degenerate clusters, yielding selective rather than universal spectral alignment.This complements mean-field and tensor-program limits by providing finite-width identities and estimates rather than asymptotic infinite-width training dynamics.Source: §2 · p. 7

Complementary frameworks and limitations

  • Relative to the flag-variety framework [19], this approach studies data-dependent gates, backward sensitivities, and covariance operators, while no random-matrix universality principle is assumed for dynamically coupled trained-network gates.The two frameworks use related commutator language but address different geometries and dynamical questions; probabilistic gate universality remains a future direction.Source: §2 · p. 7

3. Preliminaries

The preliminaries define the scalar-regression setting, feedforward network and backpropagation operators, then organize forward covariances, sensitivities, spectral projectors, and commutators for subsequent analysis. Gradient-flow conventions and dissipation connect activation correlations with backward loss sensitivities without assuming commutator decay.

3. Preliminaries

  • The framework studies scalar-valued population regression, with empirical analogues obtained by replacing distributional expectations with finite averages.The scalar-output restriction simplifies notation, while the operator identities extend to empirical settings and vector-valued outputs.Source: §3 · p. 7

3.1. Feedforward networks.

  • The model is a bias-free depth-L feedforward network with layerwise linear maps followed by componentwise, locally Lipschitz activations.ReLU is handled separately from smooth-activation theory, and fixed layerwise normalizations can be absorbed into weights and covariance operators.Source: §3.1 · p. 8

3.2. Backpropagation and Jacobian transport.

  • Backpropagation is represented by adjoint layer Jacobians, whose products transport output gradients backward through the network.At nondifferentiable activation points, a fixed measurable derivative selection is used, including σ′(s) = 1{s>0} for ReLU.Source: §3.2 · p. 8

3.3. Sensitivity and covariance operators.

  • The analysis compares gate-covariance, sensitivity-covariance, and AGOP-NFM commutators, treating them as compatibility diagnostics whose behavior need not decay during training.These operators are linked through congruence transport between backward sensitivities, forward covariances, and neural feature matrices.Source: §3.3 · p. 9

3.4. Spectral projectors.

  • Spectral projectors and eigenvalue gaps provide coordinates for localizing activation-covariance and sensitivity-covariance incompatibility.Projector definitions depend on eigenbasis choices within degenerate spectral clusters.Source: §3.4 · pp. 9–10

3.5. Imbalance and fluctuation operators.

  • Adjacent-layer imbalance, centered sensitivity fluctuations, and gate commutators form three local sources, while downstream commutators are transported recursively.Imbalance is conserved in deep linear networks but acts as a nonlinear source; zero imbalance implies adjacent covariance commutation, but not conversely.Source: §3.5 · p. 10

3.6. Gradient flow.

  • Gradient flow is formulated for locally absolutely continuous trajectories, with ReLU gate variation handled by finite-increment estimates rather than time differentiation.Its dissipation identity couples forward activation correlations with backward loss-sensitivity correlations.Source: §3.6 · pp. 10–11

4. The internal commutator hierarchy

The hierarchy distinguishes feature-side alignment from the internal commutator that generates it, then decomposes internal incompatibility into transported and local sources. Its evolution depends on cancellation, imbalance geometry, and sensitivity fluctuations, none of which is guaranteed to decay under training.

4. The internal commutator hierarchy

  • The internal commutator is more informative than the feature-side observable, which applies singular-value-weighted congruence transport that can suppress components, erase covariance-null directions, and make inversion unstable.Feature-side alignment therefore does not uniquely determine internal geometry, even when the weight matrix has full row rank.Source: §4 · pp. 11–12
  • The hierarchy propagates incompatibility backward from deeper layers while local generation begins with imbalance and gate geometry; it remains insensitive to mixing within degenerate or nearly degenerate covariance eigenspaces.Buffered localized energies are required for finer resolution, and terminal-layer commutators vanish.Source: §4 · pp. 12, 14
  • The exact internal recursion separates downstream transport from adjacent-layer imbalance, pointwise sensitivity fluctuations, and nonlinear gate-covariance interactions, whose skew-symmetric contributions can substantially cancel.A small internal commutator may reflect either individually small sources or cancellation among large sources; no component is asserted to decay during training.Source: §4 · pp. 12–14

4.1. Evolution of the adjacent-layer imbalance.

  • In nonlinear networks, full matrix balancedness generally fails: only anisotropic imbalance components affect the sensitivity commutator, and balanced initialization can still generate off-diagonal imbalance.Deep-linear networks conserve the full imbalance without regularization, whereas quadratic weight decay yields exponential decay; gradient-flow risk dissipation alone does not ensure zero limiting imbalance.Source: §4.1 · pp. 15–16

4.2. Sensitivity fluctuations across the data distribution.

  • Sensitivity fluctuation vanishes exactly when downstream sensitivity is almost surely constant, but at earlier layers its size has no general monotonicity law and must be assumed small or measured empirically.Poincaré estimates can control smooth sensitivities, whereas ReLU activation-boundary jumps make pairwise activation-pattern variation the more robust diagnostic.Source: §4.2 · pp. 16–17

5. Spectral localization and covariance geometry

The section develops spectral localization for covariance-space commutators, tracks covariance-subspace motion under training, and characterizes singular-value-weighted transport to feature geometry. These estimates describe resolution and stability but do not imply commutator decay or faithful internal alignment.

5.1. Common and buffered spectral localization.

  • Common and buffered energies localize gate and sensitivity mixing across covariance spectral clusters, with buffers excluding only direct coupling to remote complements.They are intrinsic near repeated eigenvalues because cluster projectors, rather than individual eigenvectors, determine the separation-sensitive localization.Source: §5.1 · pp. 18–19

5.2. Motion and stability of covariance subspaces.

  • Covariance-subspace projectors evolve according to cross-cluster motion controlled inversely by the separating spectral gap, without requiring eigenvalue simplicity within clusters.Weight decay changes covariance eigenvalues but not eigenspaces directly, while ReLU gate discontinuities require switching or finite-time increment estimates; the resulting bounds establish stability, not monotonicity.Source: §5.2 · pp. 20–21

5.3. Transport to feature-side geometry.

  • Feature-side commutators are spectrally filtered by λℓ,iλℓ,j, suppressing low-variance mixing and allowing apparent feature alignment despite persistent internal sensitivity mixing.Thus feature-side alignment can vanish without full internal alignment, because covariance-null or low-variance directions may be invisible after transport.Source: §5.3 · p. 23
  • The localized quantities measure distinct aspects of the same geometry and should be compared rather than identified; none is forced to decrease during training.Pointwise AGOP alignment additionally requires control of localized sensitivity fluctuations, while higher-order congruence changes spectral weighting without creating a scale-independent monotone hierarchy.Source: §5.3 · p. 23

6. Activation geometry and gate switching

Activation gates and averaged sensitivities share covariance localization but evolve differently: smooth gates follow preactivation dynamics, while ReLU gates switch between diagonal projections. The resulting estimates control gate variation and spectral-region coupling, but do not by themselves imply commutator decay or vanishing localized energies.

6. Activation geometry and gate switching

  • Gate dynamics differ fundamentally: averaged sensitivities arise from backward transport and averaging, whereas pointwise gates depend nonlinearly on preactivations and have no intrinsic alignment mechanism.For smooth activations gates have classical velocities; ReLU gates evolve through switching diagonal projections.Source: §6 · p. 24

6.1. Smooth activation geometry.

  • Smooth gate motion is driven by local weight velocity and activation dynamics transported from preceding layers, with variation controlled through preactivation variation.These bounds have no preferred sign and therefore do not imply decay of the gate-covariance commutator.Source: §6.1 · p. 24

6.2. ReLU regions and gate switching.

  • For ReLU, activation interfaces are contained in zero-preactivation sets, and gate-covariance energy couples covariance spectral separation to activation-region overlap without differentiating the gate.The symmetric-difference formulation remains valid without the regularity needed for classical interface-velocity descriptions.Source: §6.2 · pp. 24–25

6.3. Localized gate-energy variation.

  • Smooth and ReLU estimates separate activation motion from covariance-subspace motion, controlling variation but not its sign; stabilized gates and projectors therefore need not yield vanishing localized energies.Alignment requires an additional mechanism controlling the limiting cross blocks.Source: §6.3 · pp. 25–26

7. Stabilization and conditional propagation of spectral alignment

The section separates stabilization from alignment and develops conditional criteria for propagating spectral alignment across localized, global, internal, and feature-side geometries. These criteria require explicit coercivity, spectral-gap, transport, imbalance, and fluctuation conditions rather than following universally from gradient flow.

7.1. Stabilization and limiting compatibility.

  • Stabilization means localized cross blocks converge, whereas alignment additionally requires their limiting buffered cross blocks to vanish.Convergence controls late-time variation but does not determine the limiting compatibility value.Source: §7.1 · p. 27

7.2. A dissipation criterion for localized decay.

  • Under uniform continuity, bounded-below risk, and the coercive dissipation estimate, localized gate or sensitivity energies are integrable and decay to zero.The coercive estimate is a geometric hypothesis: gradient flow and bounded gate operators alone do not imply it, even when risk converges.Source: §7.2 · pp. 27–28

7.3. From localized to global alignment.

  • Decay of every unbuffered localized energy implies global commutator decay under bounded covariance scale, while the converse requires uniformly separated spectral interfaces.Without the scale and spectral-gap hypotheses, these implications need not hold.Source: §7.3 · pp. 28–29

7.4. Propagation through the internal hierarchy.

  • Gate alignment alone does not propagate through the internal hierarchy: transported incompatibility, adjacent-layer imbalance, and sensitivity fluctuations must also vanish.Under these independent requirements, backward induction yields internal alignment; unregularized gradient flow may leave imbalance at a nonzero limit.Source: §7.4 · p. 29

7.5. Consequences for feature geometry.

  • Internal alignment yields feature-side alignment under bounded transport, but the converse can fail when small or clustered singular values suppress internal directions or directions lie in ker W^T.Thus the section establishes conditional propagation rather than universal alignment along training trajectories.Source: §7.5 · pp. 29–30

8. Conditional coercivity and Lyapunov principles

Selective alignment follows only under conditional geometric coercivity: either defects are bounded by excess risk or sensitivity dynamics intrinsically damp selected cross blocks. These mechanisms yield conditional decay results while leaving generic emergence from gradient flow unestablished.

8. Conditional coercivity and Lyapunov principles

  • Risk dissipation and even a PL inequality control objective decay rather than localized geometric defects, so the proposed coercivity mechanisms remain conditional rather than generic consequences of gradient flow.The static principle applies to localized gate and sensitivity defects, whereas the dynamic principle targets sensitivity energies because ReLU gates change through switching events.Source: §8 · p. 30

8.1. Variation and risk-based transfer.

  • Under a geometric error bound q_i(t) ≤ K_iE(t) and a PL inequality, E(t) and selected sensitivity defects decay as e^-2µt, with the same transfer available for gate defects.This alignment conclusion comes from the error bound, not from the production estimate; the condition is restrictive and requires compatibility between minimizing components and selected spectral blocks.Source: §8.1 · pp. 32–33

8.2. Intrinsic transverse damping.

  • Intrinsic transverse damping yields exponential decay of excess risk and all selected sensitivity energies, without assuming each defect is controlled by excess risk.The negative damping term remains effective as risk dissipation weakens, but the required estimate is model-dependent and does not follow from controlling Qℓ’s variation.Source: §8.2 · pp. 34–35

8.3. A linear consistency example.

  • A trainable two-layer linear example satisfies the Lyapunov assumptions with µ = 1, κ = 3, and c = c∗, while a nonzero initial defect decays exponentially.The example verifies consistency of the conditional mechanism, not generic nonlinear feature alignment; part of the decay occurs because Q(t) collapses.Source: §8.3 · p. 36

8.4. Consequences for the other alignment levels.

  • Internal sensitivity alignment transports to AGOP-NFM alignment under a uniform bound on ∥Pℓ(t)∥op, but the converse can fail through small singular values or a kernel of Wℓ.Gate alignment requires a separate error-bound or damping argument, and ReLU needs a bounded-variation formulation with an explicit switching remainder.Source: §8.4 · pp. 36–37

9. Analytic examples

Analytic examples show that gate, internal, and feature-side commutators capture distinct, layer-dependent geometries: they can decouple, cancel, be hidden by transport or parametrization, and evolve nonmonotonically. These examples therefore rule out universal alignment or decay interpretations.

9.1. A two-neuron hidden layer.

  • For two ReLU neurons, gate compatibility has two distinct zero mechanisms—orthogonal weights or coincident activation half-spaces—and is therefore not monotone in neuron correlation.Antiparallel neurons produce complementary gates and maximal gate energy for fixed norms.Source: §9.1 · p. 37

9.1.1. Internal and feature-side alignment.

  • Gate alignment can coexist with nonzero internal alignment, while rank collapse can hide internal incompatibility and well-conditioned feature transport transmits it quantitatively.With orthogonal first-layer weights, gate energy vanishes even when the internal commutator is nonzero under unequal sensitivities.Source: §9.1.1 · p. 38

9.1.2. Exact cancellation of local sources.

  • Nonzero local sensitivity and gate sources can cancel exactly, producing internal and feature-side alignment without either mechanism being small.The cancellation is exhibited in a two-dimensional Gaussian ReLU construction.Source: §9.1.2 · p. 38

9.1.3. Dependence on a function-preserving parametrization.

  • Function-preserving positive-homogeneous rescaling can create exact internal and feature-side alignment, showing that commutators describe parameter geometry relative to a chosen parametrization.This alignment changes no represented function and does not make the quantities arbitrary.Source: §9.1.3 · pp. 38–39

9.1.4. Transient growth along a constrained gradient flow.

  • Along a convergent constrained gradient flow, gate energy can first increase and then decrease, because covariance coupling initially grows before gate disagreement later dominates.The trajectory converges to exact activation alignment, but gate energy is not a Lyapunov function.Source: §9.1.4 · p. 39

9.2. A minimal depth-two hierarchy.

  • A depth-two ReLU hierarchy creates new nonlinear incompatibility at deeper layers despite first-layer gate alignment, while isotropy can mask deeper incompatibility and nonzero sources can cancel again.The model separates gate disagreement from sensitivity imbalance and shows that incompatibility may be generated anew from layer to layer.Source: §9.2 · pp. 40–41

9. Analytic examples

  • Across the examples, commutator alignment is neither universally coupled nor monotone: cancellation, singular-value suppression, parametrization, and transient growth can all obscure incompatibility.Deep linear networks provide a reference where activation and nonlinear gate sources vanish identically.Source: §9 · p. 37, §9.2 · p. 41

10. Numerical experiments

Numerical experiments examine how risk reduction, width, depth, parametrization, and transport–imbalance interactions shape gate, internal, and feature geometries. Across synthetic and benchmark trajectories, the three commutators remain distinct, with small observed defects arising through structured cancellation rather than universal decay.

10.1. Experimental setting and observables.

  • Experiments use bias-free, fully connected ReLU networks with scalar linear output; the capacity experiment fixes H = 4 and uses widths 16, 32, 64, with final checkpoint T = 100.Undefined values are omitted from corresponding summaries rather than replaced by zero; such results are called active-seed summaries, and dead-layer placeholders are not interpreted as alignment or cancellation.Source: §10.1 · pp. 41, 42, 44

10.2. Risk reduction does not force global alignment.

  • Reducing risk from 10−1 to 10−3 leaves gate defects essentially constant while input internal and feature defects increase before stabilizing, providing no evidence that risk minimization alone forces global alignment.At r = 10−3, input internal means are 0.0646, 0.0805, and 0.0793 across H = 2, 4, 6, while feature means are 0.0211, 0.0199, and 0.0201.Source: §10.2 · pp. 44–45, Figure 1 · p. 45

10.3. Depth propagation and systematic source cancellation.

  • At matched risk, interior cancellation keeps the internal commutator near thirty percent of summed source magnitudes, with transport and imbalance interactions ranging from −0.716 to −0.649.Imbalance dominates interior source energy, transport rises to approximately 20%-25%, and fluctuation contributions decrease toward the output.Source: §10.3 · p. 46, Figure 2 · p. 46

10.4. Width dependence and the localized Lyapunov question.

  • Across tested widths 16, 32, and 64 at r = 10−3, internal and feature defects decrease, whereas the gate defect does not; localized defects can diverge from declining risk.For buffered cutoff (k, β) = (1, 1), median paired growth factors are 2.15, 1.96, and 1.79 at widths 16, 32, and 64. Risk dissipation alone cannot supply the conditional Lyapunov hypotheses.Source: Figure 3 · p. 47, §10.4 · pp. 47–48, Table 4 · p. 48

10.5. Replication on scalar-regression benchmarks.

  • On Concrete and Energy, gate defects remain about 0.24 while internal and feature defects are about 0.03, and both benchmarks reproduce strong cancellation dominated by negative transport–imbalance interaction.Layer-1 means are 0.2365, 0.0320, and 0.0281 for Concrete, and 0.2397, 0.0327, and 0.0302 for Energy; imbalance carries approximately half the source energy.Source: §10.5 · p. 48, Figure 4 · p. 49

10.6. A function-preserving imbalance intervention.

  • Function-preserving positive-homogeneous rescaling sharply redistributes source geometry: at c = 1/2 transport and imbalance nearly cancel, whereas at c = 2 transport contributes 0.21% and imbalance plus gate sources dominate.The initialized function and activation gates are unchanged, but the intervention also changes covariance scales and gradient parametrization, so differences cannot be attributed exclusively to ∆1.Source: §10.6 · pp. 49–50
  • Function-preserving rescaling produced a persistent geometric effect: Q(c) was 1.219 at fixed time and 1.235 at matched empirical risk 10^-3.The effect persisted after matching risk, although source decomposition was more stable than the seed-dependent endpoint defect; Figure 5 shows trajectories ranging from near-exact cancellation to nearly absent transport.Source: §10.6 · p. 50, Figure 5 · p. 51

10.7. Validation and interpretation.

  • Exact finite-sample identities validated implementation, with normalized four-source and feature-transport residuals never exceeding 1.25 × 10^-15 and 6.54 × 10^-13, respectively.These residuals were implementation checks, not empirical evidence for alignment; spectral and projector checks also passed.Source: §10.7 · p. 51
  • All alignment and source claims are finite-time ensemble observations restricted to the tested architectures, widths, datasets, learning rate, normalizations, and optimization horizon.No run reached exact interpolation, and neither step-size refinement toward continuous gradient flow nor systematic alternative-normalization comparisons were included.Source: §10.7 · p. 51
  • Risk reduction did not force commutator defects to decay: width reduced internal and feature-side defects but not gate defects, while negative transport–imbalance interactions persisted.These findings support structured cancellation rather than universal Lyapunov decay as an explanation for small observed commutators.Source: §10.7 · p. 51

11. Concluding remarks

The paper presents a coupled hierarchy of finite-width network geometries with exact commutator identities, while showing that alignment requires mechanisms beyond risk reduction. It identifies cancellation, filtering, and damping as decisive factors and outlines extensions for establishing verifiable localized damping.

Concluding remarks

  • The hierarchy links gate, internal, and feature-side geometry through four-source commutator recursion, buffered spectral localization, and singular-value-weighted transport.The middle commutator separates downstream transport, adjacent-layer imbalance, sensitivity fluctuations, and nonlinear gate-covariance interactions.Source: §11 · p. 52
  • Risk reduction does not guarantee monotone alignment because stabilization differs from decay, while cancellation and singular-value filtering can produce small net commutators.Analytic and finite-time results show separated geometries, exact cancellation, transient growth, and parametrization dependence; sufficient decay mechanisms are geometric error bounds and intrinsic transverse damping.Source: §11 · p. 52
  • The central open problem is to derive verifiable model-dependent conditions for localized damping with a non-collapsing sensitivity operator.Proposed extensions include stochastic training, wide-network feature-learning limits, and implicit selection among internal geometries compatible with interpolation.Source: §11 · p. 52

Related papers

Loading 2608.22910v1…