PaperlayerView exact revision on arXiv

Source-linked AI summary

Asymptotic Behavior of Transformers

Katy Blumer, Da-Cheng Juan, Jon Kleinberg, Ravi Kumar, Cyrus Rashtchian, Andrew Tomkins

arXiv:2610.19358v1cs.LGcs.AImath.STstat.TH

TL;DR

The paper asks how a fixed transformer behaves when it generates for an arbitrarily long time, rather than how performance changes with context length. It studies empirical state through ODEs and finds that sufficiently high temperature ensures convergence for each fixed model, while no universal finite temperature guarantees convergence across models.

  • Problem

    Most theoretical analyses fix sequence length or study context-window changes, leaving the behavior of a fixed transformer as generated tokens tend to infinity as a distinct question.

    Source: §1 · p. 1

  • Method

    The paper models slowly changing empirical token or cache distributions with limiting ODEs and analyzes how temperature affects their dynamics.

    Source: §3.2 · p. 15, §6 · p. 21

  • Results

    For each fixed transformer, sufficiently high temperature yields convergence to a unique limiting state, but at every finite temperature some transformer can fail to converge.

    Source: §1 · p. 3, §6 · pp. 21–22

  • Takeaways & Limitations

    Temperature controls long-run stability model by model, but no universal finite temperature guarantees convergence for all transformers.

    Source: §6 · p. 22

    The analyzed multi-layer setting assumes fixed depth and heads, fixed-width local windows, bounded reachable states and logits, and uniformly Lipschitz maps; temperature is bounded below by a positive constant.

    Source: §3 · p. 9, §3.1 · p. 12

Abstract

from arXiv · show

In this paper we study the behavior of transformer networks generating text over long periods of time. We show that even a three-token, single-layer attention-only transformer can fail to converge. We then study a rich class of transformers with arbitrary finite depth, layer normalization, MLPs, RoPE encodings, and an arbitrary mix of local and global layers. Every transformer in this class has a softmax temperature threshold above which its empirical cache distribution converges to a unique limit, independent of initial conditions. We show further that, while each transformer has a temperature threshold, the supremum of this set is infinite, so there is no global minimal temperature that forces all transformers to converge.

1 Introduction

The paper shifts attention from fixed-length analysis to the long-run behavior of a fixed autoregressive transformer, motivated in part by models that may remain in a generated state. It describes this behavior through deterministic ODEs and shows that high temperature ensures convergence for each fixed model, but no single finite temperature works for all models.

1 Introduction

  • For a single-layer attention-only model without positional encodings, MLPs, or normalization, the empirical token distribution summarizes history and tracks an ODE almost surely.Freezing the history makes the model a finite-state Markov chain whose steady state shifts as the empirical distribution drifts.Source: §1 · pp. 1–2
  • The empirical cache process almost surely tracks a deterministic ODE over fixed intervals of a rescaled time, reducing long-run generation to dynamics on fixed-dimensional cache distributions.The cache changes by only 1/(t + 1) per step, motivating the s-time scale j = log t + O(1).Source: §1 · p. 3
  • For every fixed transformer, sufficiently high temperature makes the cache ODE contractive and the empirical cache converges almost surely to a unique globally attracting fixed point.The convergence threshold depends on the model; no uniform threshold applies across the class.Source: §1 · p. 3
  • At every fixed finite temperature, some simplified three-token, single-layer attention-only transformer fails to converge, so no universal finite temperature guarantees convergence across models.Scaling its output embeddings by the temperature leaves its next-token probabilities unchanged.Source: §1 · p. 3

1.1 Background

  • Stochastic approximation relates accumulated-time ODE tracking to the process's limiting set, and a globally attracting fixed point yields convergence.The paper applies this framework to empirical token and cache distributions (Bena¨ım, 1999).Source: §1.1 · p. 4

2 Single-Layer Transformer

The single-layer analysis shows that sufficiently high softmax temperature makes the empirical token distribution converge to a unique limit, while a two-token example illustrates temperature-dependent limiting behavior. The proof tracks token frequencies with an ODE and bounds how its stationary distribution changes with history.

2 Single-Layer Transformer

  • Without positional encodings, the single-layer model summarizes its history by the empirical token distribution, and next-token probabilities define a transition matrix.The model omits layer normalization and MLPs and uses one attention head.Source: §2 · p. 4

2.1 Limiting Dynamics

  • The empirical token distribution tracks an ODE almost surely, whose limiting set is internally chain transitive.Its drift is the stationary distribution of the Markov chain induced by freezing the history, minus the current distribution.Source: §2.1 · p. 5

2.2 Jacobian

  • The stationary-distribution map has Jacobian norm O(1/τ) at sufficiently high temperatures, making the map a contraction above a threshold.The contraction yields a unique globally attracting fixed point of the ODE.Source: §2.2 · p. 8
  • For every τ > τc, the empirical token distribution converges almost surely to a unique fixed point; as τ → ∞, that point approaches uniformity.At τ ≤ τc, uniqueness of the limiting set is not guaranteed, and behavior may depend on model parameters.Source: §2.2 · p. 8

2.3 A Simple Example

  • In the two-token example, τ > a gives a unique attracting fixed point, whereas τ < a yields two symmetric attracting fixed points.The paper also notes that the historical token distribution in these models converges for every positive temperature, though its limit may be random.Source: §2.3 · p. 9

3 Multi-Layer Transformer

The section analyzes fixed-width, fixed-depth transformers by representing their unbounded history with an empirical cache distribution and examining recent-token behavior with a frozen cache alongside slower cache evolution. It establishes that, above an architecture- and weight-dependent finite temperature threshold, the cache-update map has a unique fixed point and the empirical cache converges to it.

3 Multi-Layer Transformer

  • A multi-layer transformer’s historical key-value vectors depend on generation context, so its unbounded history is represented by an empirical distribution over fixed-dimensional vectors; local layers depend only on the latest W = 1 + Lloc(w − 1) tokens.The section also shows that averages along RoPE phases converge to the uniform average on Hθ, uniformly in the starting phase.Source: §3 · pp. 9, 11

3.1 Frozen Cache Dynamics

  • With the global cache fixed, every recent-token state is reachable and the process converges to a unique phase-dependent distribution, independently of its initial state.Over blocks long enough to forget the initial W-gram but short enough that the cache changes little, this description remains accurate; RoPE phase dependence can be averaged over a long block.Source: §3.1 · pp. 12–13

3.2 Slow Cache Dynamics

  • On slow time s = log t + O(1), the empirical cache tracks the dynamics η′ = Γ(η) − η almost surely, and this equation has a unique solution for each initial cache distribution.Γ(η) is the average distribution of the next cache vector after recent tokens adjust to the frozen cache and RoPE phase is averaged; the result does not establish that the dynamics converge.Source: §3.2 · pp. 14–15

3.3 High-Temperature Convergence

  • The high-temperature analysis begins by establishing a contraction result for expected next-cache states, using a weighted cache metric and the O(1/τ) sensitivity of softmax.The supplied passages state that the contraction bound has a term A(δ) that tends to zero as δ tends to zero, but do not include the complete result.Source: §3.3 · pp. 16–17
  • In this high-temperature regime, the empirical cache converges almost surely to η* as time grows.The proof establishes convergence in d1,δ and uses its equivalence with d1.Source: §3.3 · p. 18
  • Above a finite threshold τc, the cache-update map Γτ is a contraction and has a unique fixed point η* for every τ > τc.The threshold may depend on the transformer’s architecture and weights.Source: §3.3 · p. 18

4 Non-Convergence

A three-token, single-layer attention-only transformer can fail to converge almost surely, and the construction extends to every positive sampling temperature. Its limiting dynamics have a unique but unstable equilibrium, while sampling noise prevents convergence to that equilibrium.

4 Non-Convergence

  • For c > 6, the unique equilibrium is the uniform distribution, but it is linearly unstable.Figure 3 depicts an attracting periodic orbit of the limiting ODE, with trajectories spiraling toward it from inside and outside.Source: §4 · p. 19, Figure 3 · p. 20
  • Because sampling noise is nondegenerate in every tangent direction, the empirical distribution does not converge almost surely when the uniform equilibrium is the only fixed point.The non-convergence argument applies the standard stochastic-approximation result for linearly unstable equilibria.Source: §4 · p. 19
  • The limiting ODE is a cyclic three-strategy logit best-response dynamic, related to rock–paper–scissors models, realized here within an explicit attention-only transformer.The paper establishes non-convergence of the transformer's empirical token frequencies.Source: §4 · p. 19
  • For every temperature τ0 > 0, a single-layer, NoPE, attention-only transformer with no MLP can have an empirical token distribution that fails to converge almost surely.Scaling its output embeddings by τ0 leaves the next-token distribution unchanged, preserving the same non-convergent dynamics.Source: §4 · pp. 19–20

5 Related Work

Related work examines attention dynamics, positional information, length extrapolation, and long-context memory. This paper instead studies the long-run generation behavior of a fixed autoregressive model.

5 Related Work

6 Conclusion

For each fixed transformer in the model, sufficiently high temperature makes the associated ODE contractive and forces convergence to a unique limiting state. The required threshold depends on the transformer, so no finite temperature guarantees convergence for all transformers.

6 Conclusion

  • For every fixed transformer in the model, sufficiently high temperature makes its ODE contractive and forces convergence to a unique limiting state.Source: §6 · p. 21
  • At every fixed finite temperature, some transformer can have a non-convergent limiting ODE, so there is no universal finite temperature guaranteeing convergence across transformers.Source: §6 · p. 22

AI use statement

  • The authors used generative AI tools to formulate and prove mathematical claims, draft parts of the paper, and identify relevant literature; they report that all LLM-assisted proofs were checked and rewritten.Source: AI use statement · p. 22

Related papers

Loading 2610.19358v1…