PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamics to decision: A mathematical theory of Lyapunov spectra and decision boundaries in deep classifiers

Shirin Panahi, Amirhossein Nazerian, Ali Pezeshki

arXiv:2609.39190v1cs.LGcs.SYeess.SYmath.DS

TL;DR

This paper asks when finite-time expansion identifies decision geometry and whether that signature survives as the observation endpoint moves backward through a classifier. It analyzes probability, logit, and hidden-representation endpoints, finding that boundary-related sensitivity depends on endpoint, training objective, and structural conditions.

  • Problem

    A decision boundary does not specify which perturbations expand, whether expansion contributes to the decision, or at what depth it arises.

    Source: §1 · p. 1

  • Method

    The paper models layerwise classifier propagation as a finite nonautonomous dynamical system and analyzes FTMLE at probability, logit, and hidden-representation endpoints.

    Source: §2 · p. 4

  • Results

    A decision boundary need not be an FTMLE maximum: probability-level sensitivity can form a minimum, logit expansion varies by loss, and fine-tuning yields 6 versus 924 exceedances with 96.96% test accuracy.

    Source: §6 · pp. 15–17

  • Takeaways & Limitations

    Boundary-relevant hidden expansion can be concentrated near the learned boundary under identifiable structural conditions and geometry-aware fine-tuning.

    Source: §1 · p. 3

    The hidden-state theorem is relative to the current logit boundary and requires a uniform r⊥ bound that forces PvDhθ(b) = 0 on B0.

    Source: §5.4 · p. 14

Abstract

from arXiv · show

A deep classifier is defined not only by the decision it produces, but also by the sequence of transformations through which that decision is formed. Treating this evolution as a dynamical system across layers provides a natural framework for asking how decision geometry emerges through depth and how far back we can trace a boundary's dynamical signature. We model a feed-forward classifier as a finite, nonautonomous discrete dynamical system, with layers playing the role of discrete time steps. We study the Finite-Time Maximum Lyapunov Exponent (FTMLE) of the data samples' dynamical trajectory through depths of the classifier. The FTMLE measures the rate of convergence/divergence of nearby trajectories. We move the observation endpoint backward from probabilities to logits and then to hidden representations. For Gaussian classes, we prove that probability-level FTMLE carries a clear geometric signature of the decision boundary, with its dominant direction aligned with the boundary normal. Moving one step backward to the logits, we prove this relationship is no longer universal but depends critically on how the classifier is trained, particularly on the choice of loss function. Moving further backward to the hidden representation, the connection becomes more conditional: boundary-related FTMLE can persist, but only under identifiable structural conditions. We propose geometry-aware fine-tuning for restructuring the classifier's hidden FTMLE, and propose conditions for guaranteed concentration of high hidden FTMLE near the decision boundary. Through our numerical results, we show the generality and validity of our theoretical results. Understanding the evolution of data samples as traveling through the layers of classifier provides a principled foundation for identifying where boundary-relevant sensitivity emerges and for developing layer-aware regularization strategies.

1 Introduction

The introduction asks when a classifier’s local expansion field reflects its decision geometry, and how that relationship changes when measured at different depths. It presents results at probability, logit, and hidden-representation endpoints.

1 Introduction

  • A decision boundary alone does not reveal which perturbations expand, whether that expansion contributes to the decision, or where it arises through depth.Extreme deformation is not automatically a region where the predicted class changes.Source: §1 · pp. 1–2
  • Prior sensitivity and boundary-geometry studies leave open when finite-time expansion reliably identifies a boundary and how that depends on the network endpoint.The cited approaches vary in the Jacobian, endpoint, and sensitivity notion considered.Source: §1 · pp. 2–3
  • For equal-prior spherical Gaussian classes, the population-optimal binary probability map has a boundary-maximal expansion profile, while multiclass facets can be normal maxima or minima.In the three-class case, a sharp transition determines whether a facet is a normal maximum or minimum.Source: §1 · p. 3
  • At the binary logit endpoint, mean-squared error on a signed target yields boundary-maximal expansion, whereas cross-entropy yields spatially constant expansion despite the same sign boundary.The contrast means a probability-level ridge need not persist at the preceding logit.Source: §1 · p. 3
  • At the hidden-representation endpoint, boundary-related expansion requires identifiable structural conditions; geometry-aware fine-tuning localizes it more strongly around the learned boundary with accuracy nearly unchanged.The introduction also proposes sufficient conditions for concentrating high hidden expansion near the boundary.Source: §1 · p. 3

2 Mathematical framework

The framework treats layerwise propagation as a finite nonautonomous discrete dynamical system and measures expansion at hidden-state, logit, and probability endpoints. It defines separate questions of directional identification and boundary-localized expansion magnitude, with statistical results developed under stated model and regularity conditions.

2 Mathematical framework

  • A deep classifier is modeled by depth-indexed maps, so each layer advances a finite nonautonomous trajectory of propagated input states.The hidden state h_k(x) is the sample propagated through the layers up to layer k.Source: §2 · p. 4
  • The FTMLE is the maximal finite-time exponential rate of divergence or convergence of nearby trajectories, computed from the largest singular value of the trajectory Jacobian.The classifier’s finite depth makes the finite-time quantity the natural one.Source: §2 · p. 4
  • The framework distinguishes whether the leading singular direction is boundary-normal from whether expansion magnitude has a normal maximum; alignment alone does not establish a boundary ridge.The three observation endpoints are the final hidden state, logit, and probability layer.Source: §2 · pp. 4–5

2.1 Statistical, probabilistic, and geometric setting

  • The statistical setting assumes differentiable, population-well-specified endpoints; proper losses recover the Bayes posterior pointwise under the stated density and continuity conditions.The equal-prior spherical Gaussian model supplies closed-form population boundary geometry, while local signed-distance and representation results have separate conditions.Source: §2.1 · pp. 5–6
  • The equal-prior Gaussian and regular-simplex assumptions support the global formulas; unequal priors shift discriminants through log-prior terms and are not silently covered.At the probability endpoint, the multiclass Jacobian composes the input-to-logit Jacobian with the softmax probability-link derivative.Source: §2.1 · p. 6

3 Input-to-probability FTMLE

At the probability endpoint, Gaussian-class FTMLE links expansion to decision-boundary geometry, with a boundary-normal leading direction. In multiclass settings, that direction can persist even when the boundary switches between a local FTMLE minimum and maximum; numerical examples agree with the theory.

3.1 Binary probability maps

  • For two equal-prior Gaussian classes with common spherical covariance, probability-endpoint FTMLE peaks on the Bayes boundary and its unique nonzero right singular direction is normal to it.The profile decays symmetrically away from the boundary and reaches α/4 at signed distance r=0.Source: §3.1 · p. 7

3.2 Regular-simplex multiclass probability maps

  • For regular-simplex Gaussian classes, the probability Jacobian admits a spectral reduction, and its leading input singular direction is normal on each smooth active facet.The section also gives the probability-endpoint finite-time Lyapunov spectrum in terms of the Jacobian singular values.Source: The identifiable pairwise logit… · pp. 7–8, §3.2 · p. 7

The identifiable pairwise logit differences obey

  • In multiclass models, a boundary-normal leading direction does not ensure a boundary maximum: the FTMLE can instead attain a local minimum there.For three classes, the facet is a strict normal minimum for 1/3 < a < ac and a strict normal maximum for ac < a < 1/2, with ac ≈0.37915287.Source: The identifiable pairwise logit… · p. 8

3.3 Numerical confirmation

  • For the multiclass example, the reported values for Jacobian-norm and unclipped FTMLE fields are 0.98766 and 0.99519, respectively; numerical curves agree with predicted minimum and maximum regimes.For the binary example, the Pearson profile correlation is 0.99787, and the learned peak is 0.50684 versus the exact peak 0.5.Source: §3.3 · pp. 9–10

4 Moving backward in time: input-to-logit FTMLE

The logit FTMLE profile depends on the training objective, even when classifiers recover the same decision boundary. In the binary Gaussian setting, MSE and cross-entropy training yield different logit sensitivity profiles.

4 Moving backward in time: input-to-logit FTMLE

  • The logit profile can differ from the probability profile because the raw logit depends on the population objective that determines it.The preceding logit can have a different expansion profile even when the output probability retains the probability-level ridge.Source: §4 · p. 10

4.1 MSE-trained binary logit

  • In the binary Gaussian setting, population-optimal MSE training produces a boundary-maximal logit FTMLE ridge because its logit is the conditional mean of signed labels.The ridge follows from the conditional mean implied by the MSE objective, not from the sign boundary alone.Source: §4.1 · p. 11

4.2 CE-trained binary logit

  • Population-optimal CE training yields an affine logit with constant FTMLE and the same decision boundary, so the boundary need not mark a logit-sensitivity maximum.The leading input direction is still normal to the boundary, but the FTMLE magnitude is spatially constant.Source: §4.2 · p. 11

5 Moving backward in time: input-to-representation FTMLE

At the hidden endpoint, total expansion can differ from the part visible to the readout, so boundary localization requires additional alignment or residual conditions. The proposed geometry-aware fine-tuning targets those conditions and concentrates hidden expansion near the learned boundary in the numerical example.

5.1 Hidden expansion, task-visible expansion, and task-alignment ratio

  • Hidden expansion includes directions that a small readout change could make visible, even when those directions currently have little effect on the logit.The task-alignment ratio compares task-visible with hidden expansion; it equals 1 when no hidden expansion is invisible to the readout.Source: §5.1 · p. 12

5.2 Attaining max hidden FTMLE on the ideal decision boundary is non-trivial

  • A classifier can have task-visible expansion maximal on the ideal boundary while hidden expansion is not even stationary there.In the example, hidden expansion reaches a global maximum on an off-boundary line.Source: §5.2 · p. 13

5.3 Sufficient ridge transfer conditions

  • Transferring a task-visible ridge to the full hidden representation is sufficient when off-boundary hidden expansion is less than its task-visible value at the boundary projection.The required control concerns directions orthogonal to the readout.Source: §5.3 · p. 13

5.4 Geometry-aware fine-tuning objectives

  • The fine-tuning objective encourages task-visible expansion to decrease away from the boundary, suppresses nuisance expansion, and penalizes changes to pretrained logits.Under the theorem's residual conditions, the learned boundary patch is a strict normal hidden-FTMLE ridge; the result is relative to the current logit boundary.Source: §5.4 · pp. 13–14

5.5 Numerical illustration

  • Among 4,813 test samples, hidden-expansion exceedances fell from 924 (19.20%) under continued CE to 6 (0.12%) after geometry-aware fine-tuning.The latter means 99.9% of assessed samples satisfy the desired hidden-expansion ordering within numerical tolerance.Source: §5.5 · p. 14, Figure 3 · p. 15
  • Fine-tuning reduced both residuals and satisfied the sampled rate condition, while off-boundary expansion declined and median linear robustness radius increased from 0.95 to 2.18.The sampled margin was 0.0904 for geometry-aware fine-tuning versus −29.0 for continued CE; test accuracy was 96.96% versus 96.94%, respectively.Source: §5.5 · pp. 14–15

6 Conclusions

The paper traces when FTMLE identifies decision geometry from probabilities back to logits and hidden representations. Its results show that boundary alignment and boundary-maximal expansion are distinct, and that interpretation depends on endpoint, training objective, and readout alignment.

6 Conclusions

  • At the probability endpoint, Gaussian binary models have boundary-maximal FTMLE with a boundary-normal leading direction, while multiclass facets can be normal maxima or minima.The multiclass result separates a boundary-normal leading direction from whether expansion magnitude peaks or dips there.Source: §6 · p. 15
  • At the logit endpoint, population-optimal MSE yields boundary-maximal FTMLE, whereas population-optimal CE yields constant FTMLE despite recovering the same boundary.The result shows that exact classification geometry does not determine the spatial sensitivity profile.Source: §6 · pp. 15–16
  • At the hidden endpoint, readout-invisible expansion can displace the maximum, while geometry-aware fine-tuning and stated conditions establish a strict normal ridge on a learned boundary patch.The conclusion cautions that FTMLE or Jacobian-norm sensitivity does not necessarily peak near a boundary.Source: §6 · pp. 16–17
  • In the tested two-moons setting, geometry-aware fine-tuning left only 6 hidden-expansion exceedances versus 924 under continued CE, while retaining 96.96% test accuracy.Approximately 99.9% of assessed samples had hidden expansion no greater than at their boundary projections, within numerical tolerance.Source: Figure 4 · p. 16, §6 · p. 17

Appendix A. Statistical preliminaries and proof of proper-loss recovery

The appendix establishes recovery of the posterior under proper probability losses and explains how population optimality yields pointwise agreement under regularity conditions. It treats binary and multiclass risks separately.

Appendix A. Statistical preliminaries and proof of proper-loss recovery

  • For binary classification, conditional squared-probability risk and cross-entropy each have the unique minimizer p = η.For cross-entropy, this includes endpoint cases under the usual extended-real convention.Source: Appendix A · p. 23
  • For multiclass classification, categorical cross-entropy and conditional Brier risk share the unique minimizer p = η.The conditional Brier risk is expressed as a constant plus the squared distance from p to η.Source: Appendix A · p. 23
  • Global population optimality and realizability imply posterior recovery almost surely, and positive density plus continuity upgrades agreement to pointwise equality with matching derivatives.The pointwise conclusion applies on an open domain where the input density is strictly positive and both maps are continuous and C1.Source: Appendix A · p. 23

Appendix B. Proofs for binary input-to-probability results

For the binary Gaussian model, the probability Jacobian is rank one and its dominant direction is the boundary normal. Its norm, and thus the depth-normalized FTMLE, is maximized at the Bayes boundary.

Appendix B. Proofs for binary input-to-probability results

  • The Bayes posterior is η(x) = sigm(αr(x)), so its 1/2-level set is B⋆ = {r = 0}.Positive Gaussian-mixture density and the stated identity establish posterior recovery pointwise.Source: Appendix B · p. 23
  • The probability Jacobian is rank one, with operator norm equal to its nonzero scalar factor and right singular direction ±n.The factor is maximized at r(x) = 0, and the increasing logarithm preserves that maximizer set.Source: Appendix B · p. 23
  • Within a tubular neighborhood, the expansion profile has a strict normal maximum at the boundary; under the stated hypotheses throughout Ω, its global maximizers are exactly r = 0.The signed-distance gradient has unit norm there, and the boundary intersection with Ω is assumed nonempty.Source: Appendix B · p. 24

Appendix C. Proofs for multiclass input-to-probability results

The multiclass probability Jacobian is governed by posterior covariance geometry, while boundary alignment depends on the posterior configuration. On smooth active facets, normal FTMLE behavior is set by a quadratic coefficient, with a fourth-order maximum at its zero crossing.

Appendix C. Proofs for multiclass input-to-probability results

  • On an active pairwise facet, the tied posteriors define the perpendicular bisector, and the leading Jacobian directions align with the class-pair boundary normal.The maximal-posterior condition restricts the tie hyperplane to the active Voronoi facet; a realizing softmax logit difference equals the pairwise log posterior ratio.Source: Appendix C · pp. 24–25
  • The probability Jacobian’s nonzero singular values are determined by the nonzero eigenvalues of G(η), which has rank C − 1 for strictly positive Gaussian posteriors.The derivation uses Dη(x) = σ−2G(η(x))V and G(η)1 = 0.Source: Appendix C · p. 24
  • At finite inputs, eigmax(G(η)) is strictly below 1/2, although it approaches 1/2 along a sequence where two posterior components tend to 1/2 each.Every Gaussian posterior component is positive, so the limiting equality is not attained at a finite input.Source: Appendix C · p. 24
  • For a smooth facet, the sign of ψij determines whether the boundary is a strict normal FTMLE maximum or minimum when ψij is nonzero.The eigenvalue continuation is even in the normal coordinate, and the positive prefactor and increasing logarithm preserve the sign-based extremum.Source: Appendix C · pp. 25–26
  • For C = 3, the quadratic coefficient changes sign at ac = (57 − 3)/12, separating the two strict regimes for 1/3 < a < 1/2.The supplied passage identifies the denominator as positive in that interval.Source: Appendix C · p. 26
  • At the coefficient’s zero crossing, the active facet is a strict fourth-order normal FTMLE maximum because the quartic coefficient is negative.The directional statement applies throughout the smooth-facet range.Source: Appendix C · p. 26

Appendix D. Proofs for input-to-logit results

The proof distinguishes two realizable binary logit losses: mean-squared error yields a tanh-shaped score, whereas cross-entropy yields a linear logit. Their input-to-logit FTMLE boundary behavior therefore differs.

Appendix D. Proofs for input-to-logit results

  • Under realizable globally optimal mean-squared-error training, the scalar logit equals the conditional target E[bY | X = x] = 2η(x) − 1.For binary labels, the conditional target is 2 sigm(αr(x)) − 1 = tanh(…).Source: Appendix D · p. 26
  • The tanh-shaped logit has the same zero set as r, and its rank-one input Jacobian has a normal direction and a uniquely boundary-maximal factor.The depth-normalized log preserves the full boundary maximizer set.Source: Appendix D · pp. 26–27
  • Under cross-entropy, the realizable optimal logit has constant derivative αn⊤, so its FTMLE is constant everywhere rather than boundary-maximal.Its zero set remains {r = 0} = B⋆.Source: Appendix D · p. 27

Appendix E. Proof for input-to-representation results

A sufficient pointwise comparison transfers localization from task-visible to full hidden expansion. Under the theorem’s uniform geometric and rate conditions, the hidden FTMLE has a strict normal ridge along the boundary patch.

Appendix E. Proof for input-to-representation results

  • Proposition 5.2 gives a sufficient pointwise condition under which localization of task-visible expansion transfers to the full hidden representation.Its comparison implies ΛH(x) < ΛH(π(x)) for the boundary projection π(x).Source: Appendix E · p. 27
  • Under the theorem’s uniform assumptions and rate condition, every admissible nonzero normal offset has lower hidden FTMLE than its boundary point.This establishes both magnitude localization and normal-direction selection on the stated smooth boundary patch.Source: Appendix E · p. 27

Related papers

Loading 2609.39190v1…