Source-linked AI summary
Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
Claudio Altafini
TL;DR
Discrete-time self-attention is natural for layer-indexed transformers, but its stability has lacked a thorough analysis. The paper classifies equilibria and studies their stability in single-head and multihead dynamics; consensus aligns with the principal eigenvector, and the predictions are confirmed on a real shared-weight transformer.
Problem
Self-attention dynamics is naturally discrete-time, yet most prior analyses use continuous-time models and a thorough discrete-time stability analysis is missing.
Source: §I · pp. 1–2Method
The paper classifies equilibria and analyzes stability in discrete-time self-attention dynamics, extending the stability analysis to the multihead case.
Source: §I · p. 2Results
Consensus aligns with the principal eigenvector of the value matrix, or the multihead layer matrix, and the discrete-time equilibrium behavior is analogous to its continuous-time counterpart.
Source: §VII · p. 15Takeaways & Limitations
The paper's predictions are confirmed by investigating a real shared-weight transformer.
Source: §VII · p. 15
Abstract
from arXiv · showhide
In machine learning, a self-attention dynamics describes the attention mechanism of transformers, and is obtained treating the layer index of a transformer as time index. Even though naturally discrete-time, this dynamical system is most often studied in continuous-time. Scope of this paper is to provide a thorough investigation of the discrete-time version of the self-attention dynamics: a complete classification of its equilibria is given, and their stability properties are studied, in both the single-head and the multihead case. We show that in both cases the main attractor of the dynamics is a consensus equilibrium aligned with the principal eigenvector of the value matrix (in the single-head case, a corresponding matrix of weights in the multihead case), even though other locally asymptotically stable bipartite equilibria can be show to exist at least in the single-head situation. The behavior is confirmed by looking at the learned weight matrices of a real transformer.
I. INTRODUCTION
The section examines equilibria and their stability in discrete-time self-attention and multiagent Oja dynamics. Both analyses focus on consensus at the principal eigenvector, while the self-attention analysis also allows other stable behaviors.
I. INTRODUCTION
- The discrete-time analysis classifies equilibria and stability, identifies principal-eigenvector consensus as the main attractor, and extends stability analysis to multihead dynamics.Single-head bipartite equilibria can also be locally asymptotically stable, and convergence is associated with the most positive eigenvalue rather than the eigenvalue of largest absolute value. Under simplifying assumptions, ALBERT weights yield consensus aligned with the layer matrix’s principal eigenvector, supporting predictions over a real transformer’s finite layer horizon.Source: §I · p. 2
II. MODEL FORMULATION
- The model uses shared single-head weights, residual connections, full attention, and L2 normalization, while omitting the feedforward network.Tokens are unit vectors on the sphere, and the discrete-time dynamics treat transformer layers as time steps.Source: §II · p. 2
III. MULTIAGENT DISCRETE-TIME OJA SYSTEM
- The multiagent discrete-time Oja system is a simpler model that retains the dynamics’ structure but removes the attention matrix.The analysis assumes a nonsingular, symmetric value matrix with a simple, positive principal eigenvalue.Source: §II · p. 2, §III · p. 3
1) consensus:
- In consensus equilibria, every token equals the same value-matrix eigenvector or its antipode.The eigenvector condition follows because the value matrix maps the shared token vector to a scalar multiple of itself.Source: A. Equilibria · p. 3, 3) polygonal equilibria · p. 3
2) bipartite consensus:
- In bipartite consensus equilibria, tokens occupy opposite points, vk and −vk, for a value-matrix eigenvector vk.Source: A. Equilibria · p. 3, 2) bipartite consensus · p. 3
A. Equilibria
- The Oja system’s equilibria include consensus, bipartite consensus, and polygonal configurations.The classification’s consensus and bipartite cases are collinear with value-matrix eigenvectors; polygonal equilibria arise when the aggregate influence is zero.Source: A. Equilibria · p. 3, 3) polygonal equilibria · p. 3
3) polygonal equilibria:
- Polygonal equilibria occur when the aggregate influence of all tokens is zero.For consensus and bipartite consensus, that influence is instead collinear with each token vector.Source: 3) polygonal equilibria · p. 3
B. Stability
- In the Oja system, principal consensus at v1 is asymptotically stable, while other consensus, bipartite, and polygonal equilibria are unstable.For almost all initial conditions, all tokens converge to v1 or −v1.Source: B. Stability · p. 3
- All bipartite consensus equilibria, xi = ±vk, are unstable for every k and every η > 0.Source: B. Stability · p. 5
- All polygonal equilibria are unstable.Source: B. Stability · p. 5
- Trajectories generically converge to an equilibrium, which must be a consensus or bipartite consensus point.Source: B. Stability · pp. 6–7
- Only consensus at the principal eigenvector is locally asymptotically stable; all other equilibria are unstable, so generic trajectories converge to that consensus point.The theorem states that all trajectories have a limit point and that generically this limit is the principal-eigenvalue consensus.Source: B. Stability · p. 7
IV. SINGLE-HEAD SELF-ATTENTION DYNAMICS
The self-attention model replaces a shared simple average with an attention-weighted average whose weights depend on the state.
IV. SINGLE-HEAD SELF-ATTENTION DYNAMICS
- State-dependent attention coefficients weight the tokens in an individual average for each agent.This replaces the simple average in the multiagent Oja model.Source: §IV · p. 7
- Because each agent has its own weighted average, the total action on an agent varies across agents.The passage denotes this agent-specific quantity as y_i(x).Source: §IV · p. 7
- The attention weights are determined by the current state, so the weighted averages change with that state.Source: §IV · p. 7
A. Equilibria
The equilibrium classification includes an additional class in self-attention dynamics because each agent can have a different total influence.
A. Equilibria
- Unlike the multiagent Oja flow, self-attention has an extra equilibrium class because y_i can differ across agents.Source: A. Equilibria · p. 7
1) consensus:
Consensus equilibria include states where every token equals the same eigenvector v_k.
1) consensus:
- The passage states that xi equals vk for every i, with k ranging from 1 through d, giving a shared value across tokens; it does not identify this state as an equilibrium.Source: 1) consensus · p. 7
2) bipartite consensus:
The equilibrium discussion includes bipartite consensus and clustering states, whose attention matrices have ranks tied to their token-group structure.
2) bipartite consensus:
- In bipartite consensus, tokens take the values +v_k or −v_k for a common eigenvector v_k.Source: 2) bipartite consensus · p. 7
4) clustering equilibria:
- In an m-clustering equilibrium, tokens take values among m vectors, and each y_i is aligned with x_i but need not align with an eigenvector of F_V.A clustering state can therefore be consensus without being aligned with an eigenvector.Source: 4) clustering equilibria · pp. 7–8
- A clustering equilibrium exists when the stated matrix involving Θ, A(x), and F_V is singular for some scalars θ_i.Source: 4) clustering equilibria · p. 8
- Flipping every token’s sign preserves a clustering equilibrium, while flipping only some tokens typically does not.Source: 4) clustering equilibria · p. 8
- At equilibrium, A(x) remains state-dependent except at consensus, while its rank is 1 for consensus, 2 for bipartite consensus, and m for m-clustering.The rank statement for m-clustering follows from the block-counting argument described in the passage.Source: 4) clustering equilibria · p. 8
B. Stability analysis
The single-head stability analysis classifies consensus, bipartite consensus, and polygonal equilibria. Principal-eigenvector consensus is stable, while other consensus points and all polygonal equilibria are unstable; some bipartite points can be stable under restricted conditions.
B. Stability analysis
- Principal-eigenvector consensus is locally asymptotically stable, while consensus equilibria associated with other eigenvectors are unstable.At consensus, the model's Jacobian has the same eigenvalues as the multiagent Oja model.Source: B. Stability analysis · pp. 8–9
- At consensus points, the self-attention model is locally identical to the multiagent Oja model.The attention matrix at consensus is uniform.Source: B. Stability analysis · p. 9
- All polygonal equilibria are unstable, including under the intrinsic perturbation used in their stability analysis.The analysis establishes instability by considering perturbations tangent to the sphere at each token.Source: B. Stability analysis · pp. 8, 13
- Some single-head bipartite equilibria can be locally asymptotically stable, but they appear rare and associated with very small attraction basins.Under the stated conditions, bipartite equilibria associated with negative eigenvalues are unstable.Source: B. Stability analysis · pp. 8, 13
V. EXTENSIONS
The paper turns to extensions of its results, with the multihead case as the main focus.
V. EXTENSIONS
- The extensions section discusses several extensions, most notably the multihead case.Source: §V · p. 13
A. Multihead case
The multihead extension combines head-specific contributions through a layer matrix and studies the resulting time-invariant dynamics. Under the stated assumptions, principal-eigenvector consensus is locally asymptotically stable, while other consensus points are unstable; bipartite stability remains unresolved.
A. Multihead case
- The layer matrix lumps together the contributions of all heads when attention is disregarded.It is formed from suitably zero-padded value matrices and the output matrix.Source: A. Multihead case · p. 14
- Under the theorem's assumptions, consensus aligned with the principal eigenvector is locally asymptotically stable, while consensus at other eigenvectors is unstable.The theorem assumes full attention and a symmetric matrix F with λ1 greater than the other eigenvalues and λ1 > 0.Source: A. Multihead case · p. 14
- The authors report no analytical or numerical evidence that bipartite consensus points are stable in the multihead system.Source: A. Multihead case · p. 14
B. Longer step size
The stability conditions distinguish consensus from bipartite equilibria: step-size restrictions for consensus are needed only in a particular spectral regime, while bipartite conditions are additional sufficient conditions.
B. Longer step size
- A consensus step-size condition is needed only when λd < 0 and |λd| > λ1; extra bipartite step-size conditions are sufficient conditions intended to simplify Jacobian eigenvalues.Locally stable bipartite equilibria appear rare, even in the single-head case.Source: B. Longer step size · p. 14
C. Asymmetric layer (or value) matrix
The asymptotic behavior can be captured without symmetric F, provided its dominant eigenvalue λ1 is real and strictly dominant. Under this condition, the behavior changes little in both single-head and multihead systems.
C. Asymmetric layer (or value) matrix
- A real dominant eigenvalue λ1 with λ1 > Re(λj) for j = 2, . . . , d is the key condition for capturing asymptotic behavior.The other eigenvalues may be any values.Source: C. Asymmetric layer (or value) matrix · p. 14
- The symmetry assumption on F, or FV in the single-head case, can be significantly relaxed.The passage identifies strict dominance of a real λ1—not symmetry—as the essential condition.Source: C. Asymmetric layer (or value) matrix · p. 14
- With strict dominance, behavior changes little for both the single-head system (1) and multihead version (13) when F is nonsymmetric.A numerical analysis of asymmetric F is carried out for ALBERT in Section VI-B.Source: C. Asymmetric layer (or value) matrix · p. 14
A. Small-scale numerical examples
The examples illustrate consensus aligned with the principal eigenvector, including in a real shared-weight multihead transformer, while single-head bipartite equilibria can be locally stable but have very small attraction basins. The paper concludes that the equilibrium and stability picture is analogous to the continuous-time case.
A. Small-scale numerical examples
- In the single-head example, the ten-token trajectory on S2 converges to consensus aligned with FV's principal eigenvector v1.The figure's lower panel shows corr(xi, v1) tending to 1 as consensus is reached.Source: A. Small-scale numerical examples · p. 14, Figure 1 · p. 15
- A single-head bipartite equilibrium can be locally asymptotically stable, but its very small basin lets a larger perturbation lead to consensus.A small perturbation is reabsorbed; analytical basin calculations are prohibitive.Source: A. Small-scale numerical examples · pp. 14–15, Figure 2 · p. 15
B. ALBERT: a shared weight multihead transformer
- Across 103 ALBERT experiments, inter-token distances vanish quickly, while alignment with F's principal eigenvector remains incomplete at T = 24 layers.Consensus can be toward v1 or −v1.Source: B. ALBERT: a shared weight multihead… · p. 15
VII. CONCLUSIONS
- The analysis finds equilibria and stability properties analogous to continuous-time counterparts, with consensus aligned to the principal eigenvector of the value matrix or multihead layer matrix.The paper reports that these predictions are confirmed using a real shared-weight transformer.Source: §VII · p. 15