PaperlayerView exact revision on arXiv

Source-linked AI summary

The Information Geometry of Representation Formation

Sanjukta Krishnagopal

arXiv:2610.34015v1cs.LGstat.ML

TL;DR

Most analyses describe trained representations, leaving their formation during training less directly measured. This paper models formation as a geometric trajectory, using Bures distance and QFI to study path movement and local sensitivity. It finds that path geometry reflects interaction order and predicts subsequent representational drift beyond endpoint displacement, and shows that representation-aware optimization can alter trajectories.

  • Problem

    Analyses largely characterize representations at checkpoints, while training-dynamics analyses more often focus on loss, parameters, or predictions, leaving a less direct view of how internal representations form.

    Source: §1 · p. 2

  • Method

    The paper models trace-normalized activation covariances as a trajectory, using Bures distance for movement and QFI for local sensitivity to controlled input variations.

    Source: §1 · p. 2

  • Results

    Higher interaction order is associated with less direct paths across controls and settings, while tortuosity predicts subsequent drift after controlling for interaction order and endpoint displacement predicts neither.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Formation geometry provides information about the computation being learned and subsequent representational stability, and representation-aware optimization can shape trajectories.

    Source: §1 · p. 4, §5 · p. 14

    The local representation-aware metric does not imply a globally shortest representation trajectory; whether such updates yield more direct paths over training is tested empirically.

    Source: §3.4 · p. 9

Abstract

from arXiv · show

Most analyses of neural representations focus on trained endpoints, leaving the process by which internal structure forms comparatively unmeasured. We instead treat representation formation itself as an object of study, asking whether the path a representation takes through training reveals information that its endpoint does not. We formalize this idea by viewing representation learning as a geometric trajectory, using Bures geometry to measure movement through representation space and quantum Fisher information to characterize local sensitivity to input factors. We find that learning concentrates local sensitivity on task-relevant factors, and that changing only the prediction target reorganizes this geometry around the corresponding visual factor. More strikingly, representation trajectories become systematically less direct as the learned computation requires higher-order interactions among input factors. This effect persists under matched-performance and training-duration controls and transfers from synthetic vector inputs to residual CNNs on rendered and natural images. Importantly, formation history contains predictive information about the stability of the learned representation: controlling for interaction order, path tortuosity predicts its subsequent drift under continued training and sensitivity to parameter perturbations, whereas endpoint displacement does not. Finally, representation-aware optimization lets us actively shape these trajectories, showing that formation geometry is not only measurable but actionable. Together, these results establish representation formation as a new source of information about learning: its geometry reveals properties of the computation being constructed, predicts subsequent representational stability, and provides a target for intervention during optimization.

1 Introduction

The paper shifts attention from trained representations to the routes by which they form, using geometry to relate local sensitivity, computational structure, and subsequent stability. It also tests whether representation-aware optimization can shape those routes.

1 Introduction

  • Checkpoint analyses describe what networks learned, but provide a less direct view of how internal representations were constructed during training.The paper motivates path analysis as complementary to endpoint comparisons.Source: §1 · p. 2
  • The framework represents hidden activations as covariance states along a Bures trajectory and uses QFI to identify controlled input variations that most strongly move the representation.The figure presents trajectory directness as tortuosity and QFI as local input-factor sensitivity.Source: §1 · pp. 2–3, Figure 1 · p. 3
  • QFI sensitivity concentrates on task-relevant factors, while changing the prediction target reorganizes local geometry around the corresponding visual factor.The supplied introduction states the task-relevant concentration finding.Source: §1 · p. 2
  • Representation-aware optimization pulls the representation geometry back to parameter space; at matched performance, RepNG reduces tortuosity by 29% relative to Adam.The paper presents this as a mechanism test, not a claim that shorter paths are universally preferable.Source: §1 · p. 3
  • Higher-order computations take less direct representation-space paths, a pattern that persists across matched-output, performance, duration, and measurement controls and transfers to residual CNNs on rendered and natural images.The paper characterizes path directness with tortuosity.Source: §1 · pp. 2–3
  • Controlling for interaction order, tortuosity predicts subsequent representation drift under continued training and parameter perturbations, whereas endpoint displacement predicts neither.This indicates that formation history carries stability information not present in endpoint displacement.Source: §1 · pp. 2–3

2 Related Work

Related work studies representation similarity, local sensitivity, representational change, and geometric optimization. This paper focuses on the trajectory of a fixed internal representation across training and asks how computation shapes it and whether its geometry can be controlled.

2 Related Work

  • RSA, CCA, SVCCA, and CKA compare representations across networks, layers, and training stages; this work instead studies a fixed representation’s trajectory across training.Prior work also treats layerwise representations as a geometric path, whereas this paper relates interaction order to tortuosity and tests matched-performance and architecture-transfer controls.Source: §2 · p. 4
  • Prior information-geometry work characterizes local stimulus sensitivity, while this paper tracks QFI through learning to examine whether sensitivity concentrates on task-relevant factors.QFI is the infinitesimal Bures metric, linking local sensitivity changes with the geometry of the full training trajectory.Source: §2 · p. 4
  • Earlier work describes representational changes such as shifts in rank, dimension, and class geometry, while transfer and continual-learning studies highlight stability–plasticity balance.The section also situates grokking as reorganization accompanying changes in generalization.Source: §2 · p. 4
  • Unlike natural-gradient approaches based on predictive-distribution changes, this framework pulls back the geometry of an explicitly chosen evolving internal representation for analysis and intervention.The paper frames its central question as how learned computation shapes formation geometry and whether that geometry can be controlled.Source: §2 · p. 5

3 Representation Information Geometry

The framework describes representation formation through checkpoint states, local sensitivity to controlled input changes, and global movement during training. Bures geometry links local QFI sensitivity with the path representations take, and its pullback provides a way to intervene on that path.

3.1 Representation States

  • A representation state is a trace-normalized covariance of hidden activations, whose geometry is invariant to common orthogonal transformations but not arbitrary invertible reparameterizations.Centering removes mean activation and trace normalization removes overall activation scale; the state is interpreted in the network’s training coordinates.Source: §3.1 · pp. 5–6

3.2 Local Geometry: Which Input Changes Matter?

  • QFI measures which controlled input directions most strongly change a representation, and its local metric is the infinitesimal Bures geometry of representation change.Its diagonal entries measure sensitivity to individual factors, while off-diagonal entries capture local coupling between factor directions.Source: §3.2 · pp. 6–7, §3.1 · p. 6

3.3 Global Geometry: Representation Trajectories

  • A representation trajectory is summarized by tortuosity, the ratio of total path length to endpoint displacement; higher values indicate less direct routes.The framework does not assume more direct paths are better, and path claims are tested under checkpoint subsampling because discrete path length depends on temporal resolution.Source: §3.3 · pp. 7–8

3.4 Pullback Geometry and Representation Natural Gradient

  • RepNG uses a pullback of the representation metric to parameter space, choosing the steepest local loss-improving direction under a damped representation-change constraint.This result is local and does not imply a globally shortest representation trajectory.Source: §3.4 · pp. 8–9

4 Experiments

The experiments test whether local sensitivity and global formation paths reveal learned task structure, predict representational stability, and can be altered by representation-aware optimization.

4.1 Local Geometry Tracks Task-Relevant Factors

  • On dSprites, changing the prediction target reorganizes QFI around the relevant visual factor; QFI also concentrates on relevant synthetic factors beyond raw activation sensitivity.For scale, relevant-factor QFI mass is 1.000 ± 0.0001; for horizontal and vertical position, it is 0.724 ± 0.037 and 0.728 ± 0.047.Source: Figure 2 · p. 10, §4.1 · pp. 10–11

4.2 Higher-Order Computations Follow Less Direct Representation Trajectories

  • For Gaussian sign-pattern tasks, mean tortuosity rose monotonically across interaction orders k=1–4: 1.261±0.042, 1.607±0.045, 1.838±0.055, and 2.049±0.058.The MLP achieved mean test accuracy above 95.7% in every condition.Source: §4.2 · p. 11
  • With binary outputs and matched performance, mean tortuosity increased from 1.014 to 1.723, 1.872, and 4.177 across interaction orders; the association also held at matched training duration.All ten within-seed slopes were positive; seed-preserving permutation tests confirmed the association.Source: §4.2 · p. 11, Figure 3 · p. 12
  • The interaction-order pattern transferred to dSprites CNNs and natural-image CIFAR-10 tasks and persisted under checkpoint subsampling and alternative distance measures.On dSprites, tortuosity increased from 3.01 to 4.93 to 7.13; on CIFAR-10 it increased from 1.54 to 4.41 to 8.13.Source: §4.2 · p. 12

4.3 Tortuosity Predicts Subsequent Representational Stability

  • Controlling for interaction order, tortuosity predicts subsequent representation drift during continued training and sensitivity to parameter perturbations, whereas endpoint displacement predicts neither.The study measured continued-training drift over 30 additional epochs and movement under small perturbations to learned weights.Source: §4.3 · pp. 12–13

4.4 Representation-Aware Optimization Changes Path Directness

  • At matched predictive performance, RepNG produces more direct paths than Adam and Euclidean SGD; its difference from output natural gradient is unresolved.Across 30 paired seeds, tortuosity was 1.895 ± 0.097 for RepNG and 2.663 ± 0.211 for Adam; under identical update norms and budgets, RepNG also outperformed output natural gradient across 20 confirmatory seeds.Source: Figure 5 · p. 13, §4.4 · pp. 13–14

5 Discussion and Conclusion

The paper argues that representation trajectories reveal computational structure and subsequent stability beyond what endpoints and accuracy alone show. It also reports that trajectories generalize across input and model settings and can be altered with representation-aware optimization.

5 Discussion and Conclusion

  • Higher interaction order produces less direct representation paths, which predict subsequent stability under continued training and parameter perturbations even after controlling for interaction order.The route provides information about the learned representation that accuracy and endpoint displacement alone do not.Source: §5 · p. 14
  • The relationship between computational structure and path geometry generalizes from Gaussian inputs and MLPs to residual CNNs on rendered and natural images.RepNG also produces more direct trajectories at matched performance; the paper identifies scaling to larger architectures and higher-dimensional parameter spaces as future work.Source: §5 · p. 14

Reproducibility Statement

The reproducibility statement identifies where to find experimental protocols, implementation details, proofs, robustness analyses, and materials for auditing and reproduction.

Reproducibility Statement

  • Section 4 reports evaluation protocols, seed counts, uncertainty estimates, stopping criteria, and controls for each principal claim.Appendix A gives architectures, data generation and preprocessing, state estimation, finite-difference, optimization, and hyperparameter-selection details; Appendices B–D give assumptions and proofs, and Appendices E–G report robustness, intervention, and stability analyses.Source: Reproducibility Statement · pp. 14–15
  • The repository and supplementary materials provide experiment code, figure-generation code, per-seed results, and numerical unit tests for auditing and reproduction.Source: Reproducibility Statement · p. 15

AI Use Statement

The authors describe the uses of language models and the checks applied to AI-assisted code. The common protocol standardizes representation-state construction and evaluation across checkpoints.

AI Use Statement

  • Language models assisted with writing, literature identification, proof checking, figure preparation, and coding, but were not an original or non-standard component of the core scientific work.The authors say AI-assisted code was checked through independent runs, robustness analyses, executable tests, and per-seed results, and that they verified the scientific claims and results.Source: AI Use Statement · p. 15

A.1 Common Protocol

  • At each checkpoint, the protocol uses the same held-out examples, subtracts the activation mean, and trace-normalizes covariance to prevent sample variation from appearing as representation movement.States and geometric operations use float64; negative eigenvalues are clamped to zero before matrix square roots, and QFI omits terms whose eigenvalue sum is at most 10^-12.Source: §A.1 · p. 19
  • Comparisons sharing data and initialization are paired by seed, with hyperparameters selected on disjoint pilot seeds and uncertainty reported using 95% Student-t confidence intervals.Unless otherwise stated, p-values use two-sided paired t-tests; test data are reserved for final evaluation when stopping uses validation data.Source: §A.1 · p. 19

A.2 Synthetic QFI Factor-Relevance Experiment

These appendices specify experimental constructions, evaluation procedures, controls, and robustness checks for the QFI, interaction-order, image-transfer, optimization, and stability studies.

A.2 Synthetic QFI Factor-Relevance Experiment

  • The synthetic QFI experiment uses mildly correlated nuisance-coordinate pairs while keeping task-relevant x1 and x2 independent, with labels given by their four sign combinations.It probes coordinate changes multiplicatively around ϕ = 0; mrel measures the share of local sensitivity in the two-dimensional relevant subspace, against an isotropic reference of 2/8 = 0.25.Source: §A.2 · pp. 19–20

A.3 dSprites Local-Geometry Experiment

  • The dSprites local-geometry study renders matched images one latent grid step above and below each base configuration, normalizing state differences by half the mean pixel distance to compare factors in equal input-arc-length coordinates.Rotation remains a varied nuisance factor rather than a prediction target.Source: §A.3 · p. 20

A.4 Interaction-Order Experiments

  • Interaction-order experiments use shared examples and data splits across multiclass sign-pattern tasks, while binary parity controls keep output dimension fixed and test trends within seeds.The matched-duration control records states through epoch 300 without accuracy stopping and tests slopes at common training budgets.Source: §A.4 · pp. 20–21

A.5 Residual-CNN and Image-Domain Transfer

  • For visual parity tasks, dSprites conditions share data, initialization, architecture, and evaluation examples, changing only the target; CIFAR-10 tasks use composites of three independently sampled images and select checkpoints closest to 70% validation accuracy.Source: §A.5 · pp. 21–22
  • In dSprites residual CNNs, tortuosity increases monotonically with interaction order in all 10 paired seeds, with a within-seed slope of 2.060 (p = 1.86 × 10^-5).For k = 3 versus k = 1, ∆τ = 4.121 (p = 1.86 × 10^-5); on CIFAR-10, path length rises from 1.86 to 4.89 to 9.00 while endpoint displacement remains 1.21, 1.11, and 1.11.Source: §A.5 · pp. 21–22

A.6 Optimizer Intervention

  • The optimizer intervention compares SGD, Adam, output natural gradient, and RepNG in a shared random 24-dimensional parameter subspace, with RepNG's pullback metric estimated from local Bures/QFI changes.Runs stop at 72% validation accuracy; scaling controls repeat the RepNG–Adam comparison in 36- and 48-dimensional subspaces.Source: §A.6 · pp. 22–23

A.7 Representational-Stability Sweep

  • The stability sweep measures post-training drift after 30 further epochs and parameter sensitivity under five perturbations, then models each outcome with interaction-order fixed effects and seed-clustered errors.Robustness sweeps vary QFI sample sizes and finite-difference steps, and recompute tortuosity using Bures, Frobenius, and additional distance measures.Source: §A.7 · p. 24, §A.8 · p. 24

B Proof of Theorem 1

The appendices establish how the paper’s local and global representation geometry is derived, interpreted, and tested for robustness. They also specify the bounds and settings under which these results apply.

B Proof of Theorem 1

  • The Bures metric’s infinitesimal quadratic form equals one quarter of the QFI quadratic form, whose leading eigenvector identifies the most sensitive local input direction.The quadratic expansion is restricted to smooth constant-rank states with tangent directions within the constant-rank stratum; first-order rank changes are excluded.Source: B Proof of Theorem 1 · pp. 24–25

C Proof of Theorem 2

  • RepNG selects the update with the greatest first-order loss decrease under a local representation-change budget.The theorem concerns a first-order loss model within a local quadratic budget, not global convergence, total path length, or finite-path optimality.Source: C Proof of Theorem 2 · p. 26

C.1 Relation to Standard Natural Gradient

  • RepNG’s metric measures changes to an internal representation, so parameter directions with similar Euclidean norms or output effects can have different costs.Standard natural gradient instead uses a metric induced by changes in the predictive distribution.Source: §C.1 · p. 26

D Operational Meaning of Bures Distance

  • Bures proximity bounds differences in normalized quadratic energy along bounded feature directions, but does not directly control labels, higher moments, or downstream accuracy.The guarantee applies to the covariance structure represented by the state.Source: D Operational Meaning of Bures Distance · p. 27

E Additional Robustness Results

  • The interaction-order ordering persists under checkpoint subsampling and three global distances, while QFI relevance is stable with sample size and increases across hidden layers.Figure 6 reports the robustness comparisons with 95% confidence intervals over ten seeds.Source: Figure 6 · p. 28

E.1 QFI Estimation and Depth

  • QFI estimates remain stable across tested finite-difference steps and sample sizes, while relevant-factor concentration increases across hidden layers.At the default step, mean relevant-factor mass rises from approximately 0.63 in the first hidden layer to 0.93 in the second; support-threshold tests also leave the reported QFI structure unchanged.Source: §E.1 · p. 28, §E.2 · pp. 28–29

E.3 Feature-Coordinate Transformations

  • Under function-preserving feature-coordinate changes, geometry is invariant to permutations and orthogonal basis changes and remains quantitatively stable under tested compensated rescalings.The geometry is not formally invariant to arbitrary invertible reparameterizations.Source: §E.3 · p. 29, Table 4 · p. 29

E.4 Checkpoint Resolution

  • Coarser checkpoint sampling lowers absolute path length and tortuosity, but preserves the complete interaction-order tortuosity ordering at every tested resolution.The tested resolutions retain every 1st, 2nd, 5th, 10th, or 20th checkpoint.Source: §E.4 · pp. 29–30, Table 5 · p. 30

E.5 Matched Training Duration

The positive association between interaction order and tortuosity persists when training duration is matched and when alternative global distances are used. These controls distinguish duration effects from the observed ordering and show that it is not specific to Bures distance.

E.5 Matched Training Duration

  • The positive interaction-order trend is present after 20 updates and remains through 300 matched updates, with significant within-seed slopes and k = 4 versus k = 1 contrasts at every tested budget.The fixed-budget control complements accuracy stopping, which matches performance but not duration; fixed budgets match duration but not performance.Source: §E.5 · p. 30, Table 6 · p. 30

E.6 Alternative Global Distances

  • All three tested distances preserve the mean tortuosity ordering across interaction order, with the complete seed-level ordering in 10/10 Bures, 9/10 CKA-angle, and 8/10 Frobenius runs.Bures is retained because its infinitesimal geometry is QFI, not because alternative distances remove the phenomenon.Source: §E.6 · pp. 30–31

F.1 Paired Optimizer Comparison

RepNG produces more direct representation trajectories than Adam and Euclidean SGD at matched predictive performance. Further controls show that this directness comparison persists across parameter-subspace dimensions and can differ from optimization speed.

F.1 Paired Optimizer Comparison

  • RepNG is more direct than Adam by 0.768 tortuosity units (p < 10−7) and more direct than SGD by 0.189 (p = 0.026), while its difference from output natural gradient is not statistically resolved.The reduction in tortuosity is driven by shorter path length rather than increased endpoint displacement.Source: §F.1 · p. 31, Table 8 · p. 31

F.2 Parameter-Subspace Dimension

  • At dimension 36, τRepNG = 1.607 and τAdam = 1.986 (p = 8.6×10−5); at dimension 48, τRepNG = 1.537 and τAdam = 1.856 (p = 3.4×10−4), with test accuracy matched within 0.5 percentage points.The Adam comparison persists as the shared parameter subspace grows.Source: §F.2 · pp. 31–32

F.3 Optimization Speed Versus Representational Directness

  • RepNG requires 65.8 steps on average to reach the validation threshold, compared with 47.3 for Adam, despite its more direct representation trajectory.Representation-space directness is distinct from conventional optimization speed.Source: §F.3 · p. 32

F.4 Update-Norm- and Budget-Matched Direction Control

  • With identical maximum update norms and a fixed budget, RepNG has lower mean tortuosity than output natural gradient (2.506 versus 3.520; paired p = 2.0 × 10−10).The difference is driven by lower cumulative path length (1.679 versus 2.639), while endpoint displacement is comparatively only slightly different (0.674 versus 0.753).Source: §F.4 · p. 32, Table 9 · p. 32

G Tortuosity and Subsequent Representational Stability

After controlling for interaction order, trajectory tortuosity predicts subsequent representational drift under continued training and parameter perturbations, whereas endpoint displacement does not.

G Tortuosity and Subsequent Representational Stability

  • Controlling for interaction order, tortuosity predicts continued-training drift (standardized β = 0.87) and parameter-perturbation drift (β = 0.51) across 120 task–seed runs.The reported associations reflect variation in formation history within task complexity rather than mean differences across interaction order.Source: G Tortuosity and Subsequent… · p. 32, Table 10 · p. 33
  • Endpoint displacement predicts neither continued-training drift (p = 0.885) nor parameter-perturbation drift (p = 0.325).Thus, formation-path information about subsequent stability is not captured by interaction order or initial-to-final displacement alone.Source: G Tortuosity and Subsequent… · p. 33

Related papers

Loading 2610.34015v1…