PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Geometry of Deep Networks: Hyperbolic Representations of Training Dynamics for Intrinsic Explainability

Ambarish Moharil

arXiv:2610.03000v1cs.LGcs.AIcs.NEphysics.data-an

TL;DR

The paper asks how neural-network structure reorganizes during training and how that self-organization shapes function, a question that single-checkpoint graph methods leave temporally underused. It embeds temporal MLP parameter graphs in the Poincaré ball with permutation-aware hyperbolic learning, and reports competitive results across classification, generalization prediction, and regression tasks.

  • Problem

    The paper investigates how neural-network weighted topology reorganizes during training and shapes function, while existing graph methods often use single checkpoints and underuse optimization traces.

    Source: §1 · pp. 1–2

  • Method

    The GTH-GMN embeds temporal MLP parameter graphs in the Poincaré ball, combining hyperbolic attention and evolving kernels while respecting neuron and snapshot permutation symmetries.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    The temporal encoder reports 95.6 ± 0.18% average test accuracy on MNIST and 80.72 ± 0.29% on Fashion MNIST, while CIFAR-10 generalization prediction reaches Kendall’s τ = 0.846 ± 0.004, modestly below neural functional baselines.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The learned embeddings show structured self-organization over training, including outward drift, layer separation, and smooth trajectories that may provide signals for intrinsic explainability.

    Source: §5 · p. 10

    CIFAR-10 generalization prediction is relatively modest compared with neural functional baselines, with Kendall’s τ = 0.846 ± 0.004.

    Source: §5 · p. 10

Abstract

from arXiv · show

Intrinsic explainability remains a challenging problem, particularly in contexts where multilayer perceptrons (MLPs) require dynamic re-training within an optimization environment. This paper investigates how MLPs and their training dynamics can be represented and studied in non-Euclidean spaces; our representation features the Poincar\'e model of hyperbolic geometry. We aim to capture the geometric evolution of their weighted topology and self-organization over time. Instead of restricting the analysis to single checkpoints---as per established measure-based explainability methods---we construct temporal \textit{parameter graphs}, i.e., snapshots over time $T$ steps of the optimization/training process for MLPs. This reflects the view that neural networks encode information not only in their weights but also in the trajectory traced during training. Drawing on the idea that many complex networks admit embeddings in hidden metric spaces where distances correspond to connection likelihood, we present a geometric and temporal graph-based metalearning framework for obtaining dynamic hyperbolic representations of the underlying neural parameter graphs. Our model embeds temporal parameter graphs in the Poincar\'e model ball, and learns from them while maintaining equivariance to within-snapshot neuron permutations and invariance to permutations of past snapshots. In doing so, the approach preserves functional equivalence over time and recovers the latent evolving geometry of the network. Experiments on regression and classification tasks with trained MLPs show strong meta-network performance, accompanied by hyperbolic temporal representations. This reveals how the network structure emerges over time under specific training environments, thus providing insights into the network's self-organization.

1 INTRODUCTION

The paper asks how neural-network weighted topology reorganizes during training and shapes function, treating this evolution as a source of intrinsic explanation. It addresses the underuse of training traces and the lack of hyperbolic parameter-graph embeddings that preserve permutation symmetry.

1 INTRODUCTION

  • The paper frames weighted topology as more than an optimization byproduct: it is a structure through which generalization, compression, and transfer learning arise.A temporal geometric view can track hierarchy, functional-module consolidation, and influence redistribution across layers.Source: §1 · p. 1
  • Earlier meta-network methods that flattened weights discarded structural relations and functional invariance, while later symmetry-aware methods often required architecture-specific tuning and Euclidean representations.These constraints make internal self-organization challenging to interpret.Source: §1 · p. 2
  • Prior graph-based meta-networks often use single checkpoints, leaving the temporal traces of optimization underused.They have performed well on INR classification and generalization prediction.Source: §1 · p. 2
  • Existing neural parameter-graph approaches commonly use Euclidean embeddings, while hyperbolic embeddings can preserve hierarchy with low distortion and had not been applied to parameter graphs with permutation symmetry.The cited closest example concerns static multilayer brain-network graphs.Source: §1 · p. 2
  • The framework embeds temporal MLP parameter graphs in the Poincaré ball, preserving within-snapshot neuron-permutation equivariance and past-snapshot permutation invariance.It couples distance-biased hyperbolic attention with evolving attention kernels and reports matching or closely approaching strong baselines across three tasks.Source: §1 · p. 2

2 RELATED WORK

Related work spans neural meta-networks, hyperbolic graph learning, and temporal graph models. The paper positions its approach at their intersection, addressing temporal parameter traces and hierarchical geometry while using evolving attention kernels to reduce memory demands.

Neural Meta Networks

  • Neural meta-networks progressed from flattened parameters to permutation-equivariant and graph-based models, but commonly use single checkpoints and Euclidean embeddings.The paper instead studies temporal parameter graphs of MLPs in hyperbolic space to capture evolving geometry and functional symmetries.Source: §2 · p. 3

Hyperbolic GNNs; Temporal Learning in Euclidean and Hyperbolic Spaces

  • Hyperbolic graph methods adapt message passing, attention, and diffusion to curved spaces, while temporal models add recurrence, temporal attention, or evolving graph encoders.The paper identifies scaling and curvature adaptation over long sequences as remaining difficulties for hyperbolic temporal models.Source: §2 · p. 3

Temporal Learning in Euclidean and Hyperbolic Spaces

  • The framework follows a kernel-evolution strategy that updates attention parameters over time rather than retaining the full sequence of hyperbolic embeddings.This reduces memory use and supports learning over long optimization traces while maintaining temporal smoothness.Source: §2 · p. 4

3 RESEARCH METHOD

The method converts MLP training traces into parameter-graph snapshots, processes them with hyperbolic attention and temporally evolving kernels, and predicts connectivity and signed edge weights. Its geometric design links distance to connection strength while refining node positions on the Poincaré ball.

3.1 TEMPORAL GRAPH CONSTRUCTION FROM MLP TRACES

  • Each MLP checkpoint becomes a graph of neurons and bias nodes connected by weighted edges, with permutation-invariant node features and a temporal sequence of snapshots.Hidden-layer reordering yields an equivariant graph structure; edge attributes encode bias and layer context without exposing weight-derived functions.Source: §3.1 · pp. 4–5

3.2 HYPERBOLIC GRAPH ATTENTION LAYER: INTUITION AND FORMULATION

  • Hyperbolic attention computes in tangent spaces and uses geodesic proximity to score neighbors, then aggregates updates in a curvature-aware manner.Einstein gyromidpoints provide manifold-consistent aggregation, and gated residual updates are projected safely inside the ball.Source: §3.2 · pp. 5–6
  • Attention kernels evolve recurrently across snapshots, generating new Q/K/V parameters without storing past embeddings.The GRU-based mechanism uses compact recurrent states to maintain temporal consistency and permutation invariance.Source: §3.2 · p. 6

3.3 SIGNED WEIGHT REGRESSION

  • The signed weight regressor separates magnitude from polarity, tying larger magnitudes to shorter hyperbolic distances and sign to directional alignment in tangent space.Its magnitude-distance relation follows a heavy-tailed power law with node-dependent scale and decay factors.Source: §3.3 · p. 7

3.4 LINK DECODER, SUPERVISION, AND OBJECTIVE

  • Training combines Fermi–Dirac link prediction, magnitude and polarity supervision, and geometric regularizers for slope, temporal smoothness, and distance ranking.Optimization uses separate Euclidean updates for kernels and regressors and Riemannian updates for node positions on the Poincaré ball.Source: §3.4 · pp. 7–8

4 EXPERIMENTAL SETUP AND RESULTS

The experiments evaluate temporal parameter-graph representations on image classification, generalization prediction, and sinusoid-parameter prediction. The supplied results passages describe task setups and table scopes, but do not provide table values.

4 EXPERIMENTAL SETUP AND RESULTS

  • The INR-classification evaluation uses 80–100 parameter-graph checkpoints per image and reports held-out accuracy for MNIST and Fashion-MNIST.Images are split 45,000/5,000/10,000 for training, validation, and testing; Table 1 covers average test accuracy ± standard error in static and temporal settings.Source: §4 · p. 8, Table 1 · p. 9
  • CIFAR-10 generalization prediction encodes MLP parameter-graph traces with last-step supervision and evaluates ranking fidelity using Kendall’s τ.Table 2 compares rank correlation between predicted and ground-truth architecture accuracies.Source: §4 · p. 9, Table 2 · p. 9
  • The sinusoid benchmark links K successive checkpoints into a temporal hyperbolic representation, then mean-pools it to predict function parameters a and b.Table 3 reports downstream-regressor test MSE for sinusoid-MLP representations.Source: §4 · p. 9, Table 3 · p. 9
  • Figure 2 visualizes a 13-layer CIFAR-10 INR’s hyperbolic parameter-graph embeddings at checkpoints t = 11 and t = 97.The embeddings are learned in 10 dimensions and projected to the Poincaré disk using PCA; nodes are colored by layer.Source: Figure 2 · p. 10

5 DISCUSSION

The discussion reports task-specific performance and interprets the evolving hyperbolic embeddings as potential explainability signals. The accompanying material introduces notation and geometric background, alongside related work on network geometry and graph learning.

5 DISCUSSION

  • 95.6 ± 0.18% on MNIST and 80.72 ± 0.29% on Fashion MNIST; the discussion attributes these accuracies to class-specific geometric structures in INR training traces.The CIFAR-10 Kendall’s τ is 0.846 ± 0.004 and is described as modest relative to neural functional baselines; sinusoid test MSE is 1.06±0.24 versus GMN’s 1.13 ± 0.08, with higher variance.Source: §5 · p. 10
  • The embeddings show layer separation and smooth trajectories; radial drift, angular separation, proximity, and magnitude–distance alignment are proposed as explainability signals.The discussion also suggests that collapsed trajectories or uniform outward drift may indicate redundancy and pruning opportunities.Source: §5 · p. 10

A NOTATIONS

  • The notation lists graph snapshots, node and edge features, temporal states, hyperbolic operations, and losses used to describe the framework.It includes layer and node-type labels, incident signed and absolute-weight signals, and the Fermi–Dirac, reconstruction, sign, smoothness, and ranking losses.Source: A NOTATIONS · pp. 17–18

B.1 RIEMANNIAN MANIFOLD

  • A Riemannian manifold uses a metric to define lengths, angles, and distances, while exponential and logarithmic maps connect tangent-space displacements with manifold points.The exponential map lifts a tangent vector along a geodesic; the logarithmic map represents a nearby point’s displacement in the tangent space.Source: §B.1 · pp. 18–19

B.2 POINCAR´E BALL MODEL

  • The Poincaré ball is presented as one model of constant-negative-curvature hyperbolic space, with geometric operations supporting optimization and representation learning.The section describes distance, exponential and logarithmic maps, and parallel transport as components of its geometric apparatus.Source: §B.2 · p. 19

C ADDITIONAL RELATED WORK

  • Related work connects network structure with hidden metric spaces and surveys graph-learning approaches in neuroscience; prior hyperbolic multilayer-network treatments are described as static.The cited discussion spans network science, neurobiological graph analysis, dynamic graph representations, and hyperbolic embeddings.Source: C ADDITIONAL RELATED WORK · pp. 20–21

D THEORETICAL PROPOSITIONS & PROOFS

The section formalizes temporal parameter traces and the allowable neuron relabelings, then establishes the encoder’s symmetry properties. It is equivariant to relabeling the current snapshot and invariant to relabeling the preceding snapshot.

Setup and definitions

  • A trace is a time-indexed family of weighted parameter graphs on a common vertex set, with each snapshot carrying its own edges and weights.The vertex set separates input, hidden-layer, output, and bias nodes.Source: D THEORETICAL PROPOSITIONS & PROOFS · p. 21

Proposition 1

Propositions 1–2

  • The encoder is equivariant to admissible permutations at the current snapshot, while the transition from the previous snapshot is invariant to its admissible permutations.The temporal kernel uses permutation-invariant summaries of the previous-step embeddings.Source: D THEORETICAL PROPOSITIONS & PROOFS · pp. 22, 24

Equivariance proof

  • Node- and edge-feature descriptors are structural and remain unchanged under within-layer hidden-node permutations that fix input, output, and bias nodes.The descriptors include layer and node type, standardized incident-weight aggregations, bias-edge status, and layer difference.Source: D THEORETICAL PROPOSITIONS & PROOFS · p. 22

Hyperbolic aggregation

  • Einstein gyromidpoint aggregation commutes with admissible node reindexing, preserving equivariance when hyperbolic neighbor values are pooled.The method computes the weighted barycenter in Klein space, maps it back to the Poincaré ball, and linearizes at the origin for the residual path.Source: D THEORETICAL PROPOSITIONS & PROOFS · pp. 23–24

E EXPERIMENT DETAILS

The experiment represents training as a sequence of signed parameter graphs and learns node embeddings in a Poincaré ball. The embeddings support edge and weight prediction while being regularized to evolve smoothly over time.

E.1 PROBLEM FORMULATION

  • The target is a sequence of node embeddings in a d-dimensional Poincaré ball whose trajectories evolve smoothly across training snapshots.The graph sequence has a fixed vertex set and time-varying undirected edges with associated signed weights.Source: §E.1 · pp. 24–25
  • The embedding objectives include predicting adjacency labels with a Fermi–Dirac decoder and regressing signed edge weights.Supervision uses positive edges and adaptively generated negative samples, making the task semi-supervised.Source: §E.1 · p. 25

E.2 ADDITIONAL DETAILS: CLASSIFICATION OF IMAGES VIA INR TRACES

The image-classification experiment converts checkpoint traces from trained image-fitting networks into temporal parameter graphs and pools their learned representations for classification. The setup specifies image-level splits and combines graph-learning objectives with temporal and ranking regularization.

E.2 ADDITIONAL DETAILS: CLASSIFICATION OF IMAGES VIA INR TRACES

  • Each image is fitted by a three-layer, width-32 tanh INR, trained with Adam for up to 100 epochs and early-stopped above 40 PSNR.The input is pixel coordinates and the output is grayscale intensity.Source: §E.2 · p. 25
  • At each checkpoint, the INR becomes a signed parameter graph with unit and bias nodes, weighted edges, and structural node and edge features.Edges link successive layers and biases to their layer; node features include layer, type, and standardized incident-weight sums.Source: §E.2 · p. 25
  • GTH-GMN pretraining uses four hyperbolic graph-attention layers paired with four EvolveGCN-H temporal kernels in a 125-dimensional ball with curvature −1.The optimization alternates Euclidean updates of attention kernels, regressors, and decoder with Riemannian node-embedding updates.Source: §E.2 · p. 25
  • The training objective combines link prediction, signed-weight regression, an exponent prior, temporal smoothness, and pairwise ranking losses.The ranking margin increases over epochs while negative-sample mining hardness is annealed.Source: §E.2 · p. 25
  • Test accuracy on image-level train, validation, and test splits evaluates an MLP classifier trained on temporally pooled checkpoint representations.Each image contributes one trace, with all its checkpoints kept in the same split; traces contain 80–100 checkpoints.Source: §E.2 · pp. 25–26

E.3 ADDITIONAL DETAILS: PREDICTING DNN GENERALIZATION FROM WEIGHTS

On CIFAR-10 checkpoint traces, GTH-GMN’s predicted generalization rankings show a strong monotonic relationship with actual performance, though its reported correlation is below NFN baselines. The section also describes temporal-trace supervision and a contrastive objective for sine-wave frequency prediction.

E.3 ADDITIONAL DETAILS: PREDICTING DNN GENERALIZATION FROM WEIGHTS

  • CIFAR-10 trial traces pair ordered parameter-graph checkpoints with per-epoch test accuracy, using last-step supervision and trial-level splits.Intermediate steps evolve the state; supervision is applied to the final-step target.Source: §E.3 · p. 26
  • Kendall’s τ = 0.846 ± 0.004 for GTH-GMN on CIFAR-10 checkpoint traces, below NFN baselines that learn end-to-end from raw weight tensors.The reported strong monotonic trend indicates meaningful structure in the learned hyperbolic embeddings, although absolute correlation lags the NFN baselines.Source: Figure 3 · p. 26, §E.3 · pp. 26–27

E.4 ADDITIONAL DETAILS: PREDICTING SINE WAVE FREQUENCY

  • For sine-wave frequency prediction, a SimCLR-style contrastive regularizer trains on augmented checkpoint subsequences, with same-subsequence views as positive pairs and other trials as negatives.Augmentations include Gaussian feature perturbations and random temporal masking.Source: §E.4 · p. 27

E.5 ADDITIONAL DETAILS: WEIGHTS VS DISTANCE RELATIONSHIPS

On CIFAR-10 INR traces, learned hyperbolic distances track both edge-weight magnitude and the strength of connections.

E.5 ADDITIONAL DETAILS: WEIGHTS VS DISTANCE RELATIONSHIPS

  • Figure 4 plots distance against absolute weight and log-magnitude using log-count hexbins on CIFAR-10 INR traces.Source: Figure 4 · p. 27
  • Large weights concentrate at small hyperbolic distances, while edge magnitudes decline systematically as distance increases.The inverse trend is approximately linear for log-magnitude, consistent with a power-law dependence.Source: §E.5 · p. 27, Figure 4 · p. 27
  • The distance–weight relationship supports the interpretation that the embeddings encode graded functional connection strengths, not only edge presence.Source: §E.5 · p. 27

F HYPERPARAMETER INVESTIGATION

The hyperparameter investigation reports architecture, data, and optimization settings used across the experiments.

F HYPERPARAMETER INVESTIGATION

  • The experiments fix curvature at |c|=1.0, sweep embedding dimension d from 80 to 125, and use four HGAT layers with four to eight heads.HGAT also uses learned head temperatures, a learned residual gate, and per-layer geodesic trust radii from 0.05 to 0.15.Source: F HYPERPARAMETER INVESTIGATION · p. 28, Table 5 · p. 28
  • EvolveGCN–H matches HGAT depth with GRU sizes 64–128; temporal windows range from K=64 to T with stride 4–8.Parameter graphs encode layer, node type, absolute and signed strength at nodes, and bias status and layer offset at edges.Source: F HYPERPARAMETER INVESTIGATION · p. 28
  • Euclidean parameters use Adam for 30–50 epochs, while node positions use Riemannian AMSGrad with matched learning rates and trust radii of 0.05–0.15.Training combines link prediction, signed-weight reconstruction, sign, exponent, temporal-smoothness, and pairwise-ranking losses; edge computations are capped and chunked for throughput.Source: F HYPERPARAMETER INVESTIGATION · p. 28

G ALGORITHMS

The algorithms construct parameter graphs from MLP snapshots and optimize their node positions on the Poincaré ball; the accompanying embeddings show layers reorganizing over training.

G ALGORITHMS

  • Algorithm 2 builds a permutation-invariant parameter graph by adding input, neuron, and bias nodes, coalescing parameter incidences, and standardizing node-strength features.Algorithm 1 updates positions on the Poincaré ball using Riemannian gradients, Adam-style moments, a geodesic trust region, and parallel transport.Source: G ALGORITHMS · p. 28
  • The objective combines Fermi–Dirac link prediction with signed weight and sign prediction, exponent regularization, temporal smoothness, and pairwise distance–magnitude ranking.The decoder uses learned radius and temperature with positive edges and dynamic negatives sampled from same-layer and uniform mixtures.Source: G ALGORITHMS · pp. 29–30
  • Training separates Euclidean updates of kernels, regressors, and decoder from a Riemannian pass that updates only node positions with detached node features.The position pass is intended to stabilize the graph-attention head and control geodesic updates near the ball boundary.Source: G ALGORITHMS · p. 30

H HYPERBOLIC EMBEDDINGS OF INRS

  • Across checkpoints, layers separate angularly, within-layer bands tighten, and high-strength nodes move inward, indicating hub formation and growing specialization.The displayed PCA projections do not preserve exact angles, but progressive stratification and shorter within-layer geodesics recur across panels.Source: H HYPERBOLIC EMBEDDINGS OF INRS · p. 30

I COMPUTATIONAL COST ANALYSIS

The cost analysis finds edge processing dominant under the studied graph regime, while temporal evolution and Riemannian refinement preserve the stated scaling.

I COMPUTATIONAL COST ANALYSIS

  • When E≫N and d is moderate, the HGAT edge term O(LHEd) dominates, making encoder cost effectively linear in edge count.Hyperbolic operations add O(d) vector calculations without changing asymptotic order relative to a comparable Euclidean graph-attention network.Source: I COMPUTATIONAL COST ANALYSIS · p. 32
  • Temporal-kernel cost is linear in window length K and independent of E; in the reported configurations, HGAT edge processing dominates it.Source: I COMPUTATIONAL COST ANALYSIS · p. 33
  • The two-phase training scheme adds constant-factor overhead over a single Euclidean pass while preserving linear dependence on E and K.The Riemannian refinement updates node embeddings without recomputing the attention stack or temporal kernels.Source: I COMPUTATIONAL COST ANALYSIS · p. 34
  • Memory retains current-window embeddings and recurrent state rather than embeddings for every checkpoint, avoiding O(TNd) storage over a full trace.Memory includes node embeddings, attention buffers, temporal states, and bounded decoder buffers.Source: I COMPUTATIONAL COST ANALYSIS · p. 34

J ABLATIONS

The ablations show that ranking, temporal smoothness, and signed weight supervision contribute to coherent hyperbolic representations in INR classification. The paper also outlines how its graph framework could extend to CNNs and Transformers and incorporate broader dynamical information.

J ABLATIONS

  • With the full objective, Kendall’s τ is 0.946 on MNIST and 0.935 on Fashion–MNIST, alongside distance–log |w| slopes of −0.617 and −0.322, respectively.The reported slopes are steepest for MNIST and moderately steep for Fashion–MNIST.Source: Figure 6 · p. 36, J ABLATIONS · p. 36, Figure 7 · p. 37
  • Removing Lrank lowers Kendall’s τ to 0.811 on MNIST and 0.797 on Fashion–MNIST, while slopes flatten to −0.473 and −0.189, weakening monotone ordering.The hexbins widen, with more local inversions between strong and weak edges.Source: J ABLATIONS · p. 36, Figure 8 · p. 37, Figure 9 · p. 38
  • The ablations assign distinct roles: ranking orders weight magnitude by distance, temporal smoothness limits checkpoint jumps, and the signed head binds geometry to weight values.The INR-trace classification setting provides 80–100 checkpoints per image and jointly activates the objective components being ablated.Source: J ABLATIONS · pp. 34, 35, 37

K FUTURE WORKS

  • The framework is proposed for CNN and Transformer parameter graphs without core-method modification, while adding global dynamical information is left as future work.Convolutional kernels and attention layers can instantiate richer graph topologies; topological conjugacy and Koopman spectral analysis are suggested as complementary signals.Source: K FUTURE WORKS · pp. 37–39

J ABLATIONS

  • Removing both Lrank and Lsmooth lowers Kendall’s τ to 0.765 on MNIST and 0.764 on Fashion–MNIST, while the Fashion–MNIST slope falls to −0.061.The expected ordering of stronger weights closer to the origin breaks down in many regions, and embeddings can jump between checkpoints.Source: J ABLATIONS · p. 36, Figure 10 · p. 38, Figure 12 · p. 39
  • Removing the signed head yields the strongest geometric degradation despite comparatively close INR accuracy; Kendall’s τ falls to 0.738 on MNIST and 0.726 on Fashion–MNIST.The distance–magnitude slope reverses to +0.125 on Fashion–MNIST, and the embedding reflects connectivity but not graded weight strength.Source: J ABLATIONS · pp. 36–37, Figure 11 · p. 38, Figure 13 · p. 39

Related papers

Loading 2610.03000v1…