PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal interpolation for deep residual self-attention networks

Sibylle Marcotte, Joan Bruna

arXiv:2610.01981v1cs.LG

TL;DR

The paper asks whether self-attention can interpolate arbitrary batches using transformations fixed independently of the input and target, a depth-based alternative to infinite-width approaches. It uses frozen attention blocks and establishes universal interpolation with two randomly sampled blocks at continuous and finite depth, while characterizing causal-mask restrictions.

  • Problem

    Universal approximation is often studied through infinite-width limits, leaving the depth-based expressivity of self-attention with fixed, reusable transformations as a question for interpolation and scaling-law research.

    Source: §1 · p. 1

  • Method

    The paper studies residual self-attention with a finite dictionary of frozen blocks, allowing task-specific choices of block, sign, order, and duration.

    Source: §1 · pp. 1–2

  • Results

    Two randomly sampled frozen single-head blocks yield universal interpolation at continuous and finite depth, independently of dataset size, on the stated noncausal domains.

    Source: §3.1 · p. 8, CONCLUSION · pp. 9–10

  • Takeaways & Limitations

    Causal masking preserves first-token linear relations, so universal interpolation is recovered on classes of batches sharing those relations rather than across the full configuration space when N > d.

    Source: §3.2 · p. 8

    The finite-depth constructions give no bound on total depth or parameter norms in terms of dataset size and architecture dimensions; noncausal domains also exclude symmetry-related input pairs.

    Source: §2.3 · p. 6, §3.1 · p. 7

Abstract

from arXiv · show

Universal approximation is a necessary qualitative property of learning architectures to benefit from scaling laws. While it is generically verified on a variety of neural architectures and random feature models, it typically involves infinite width limits. In this work, we focus on deep self-attention models and consider instead the `dual' regime, where approximation power is enabled entirely by depth, and featuring strong parameter sharing across layers, motivated by recent models such as the Looped Transformers. More specifically, we ask whether one can find a predefined finite set of parameters, each defining an attention block, such that the resulting finite set of transformations can map any collection of $N$ sequences of $n$ tokens to any other collection of $N$ sequences of $n$ tokens. Crucially, these transformations are \emph{fixed independently of the input and output} collections: only the order in which the blocks are applied, their signs, and their durations depend on the particular interpolation task. Our main result establishes it for residual softmax attention using only two frozen single-head blocks with Gaussian-initialized projection matrices. The result holds at both continuous and finite depth. We also characterize the restrictions imposed by causal masking and establish corresponding universal interpolation guarantees.

1 INTRODUCTION

The introduction frames fixed-generator interpolation as a depth-based route to characterize transformer expressivity, distinct from fitting or approximating functions with data-dependent parameters. It presents results for frozen attention blocks, including limits and guarantees under causal masking.

Introduction

  • Depth-based expressivity is posed as a stepping stone toward characterizing transformer architectures for studying scaling laws, complementing approaches based on infinite-width limits.The introduction asks whether depth can serve as the instrumental variable for universal approximation, analogously to deep versions of random feature models.Source: §1 · p. 1
  • The paper asks whether a finite family of attention blocks fixed before seeing the data can achieve exact interpolation by varying block choice, scalar gain, order, and duration.The paper calls this property universal interpolation.Source: §1 · p. 1
  • Parameter reuse in looped and recurrent-depth transformers motivates asking how much expressive power comes from composing fixed transformations.These architectures repeatedly apply a small number of blocks rather than using independent parameters at every layer.Source: §1 · p. 2

Related work

  • Prior exact-interpolation and universal-approximation results allow transformations or parameters to depend on the task, unlike the fixed transformations studied here.Earlier fixed-family interpolation results for attention had not established the result for standard softmax attention.Source: §1 · p. 2

Our contributions

  • Two frozen random single-head noncausal attention blocks suffice for universal interpolation on a connected domain of full measure, while causal masking preserves first-token relations and restricts interpolation.For causal attention, exact interpolation fails when N > d, but universal interpolation holds on classes whose input and target batches share the same first-token relations.Source: §1 · p. 3
  • The contributions also transfer interpolation criteria from continuous to finite depth and establish causal-attention guarantees on classes defined by shared first-token relations.For causal attention, the first-token update is linear, so relations among batch first tokens persist at every depth.Source: §1 · p. 3

2 EXACT INTERPOLATION AND UNIVERSAL INTERPOLATION

This section formulates interpolation for residual networks through continuous-depth controls and criteria based on the rank of update fields and their Lie brackets. It then explains how these criteria support finite-depth interpolation and what bounds they do—and do not—provide.

2.1 FROM RESIDUAL LAYERS TO CONTINUOUS-DEPTH NETWORKS

  • A self-attention batch is represented by ordered, vectorized token sequences, preserving tokenwise input–output correspondences that empirical-measure descriptions omit.The continuous-depth residual update is treated as a forward Euler discretization with parameters shared across inputs and potentially varying with depth.Source: §2.1 · p. 3

2.2 EXACT AND UNIVERSAL INTERPOLATION

  • Exact interpolation permits task-dependent updates, whereas universal interpolation requires a finite update dictionary fixed across admissible input–target pairs.For universal interpolation, fitting changes the selected signed updates, their order, and durations; the selected parameters may depend on N, but the architecture remains fixed.Source: §2.2 · p. 4

2.2.1 CRITERIA FOR EXACT AND UNIVERSAL INTERPOLATION

  • Full Lie rank of the lifted updates and their iterated brackets yields universal interpolation when a single finite parameter set supplies locally spanning fields throughout the connected configuration domain.The selected subset may vary with the configuration, but the dictionary itself is fixed.Source: §2.2.1 · p. 5

2.2.2 COMPUTING THE LIE-ALGEBRA RANK

  • The rank procedure iteratively brackets the original generators with fields built so far and evaluates their span, using automatic differentiation and Jacobian–vector products for nested brackets.A rank equal to the batch-space dimension certifies full Lie rank at that configuration.Source: §2.2.2 · pp. 5–6

2.3 EXACT AND UNIVERSAL INTERPOLATION AT FINITE DEPTH

  • Full Lie rank suffices for finite-depth interpolation, while direct spanning permits a final exact correction using at most D = Np additional residual layers.The correction uses D independent update directions to cancel a sufficiently small Euler-discretization error near the target.Source: §2.3 · p. 6
  • The finite-depth constructions provide no bound on total depth or parameter norms in terms of dataset size and architecture dimensions.The D = Np bound applies to the final correction under direct spanning, not to the full interpolation trajectory.Source: §2.3 · p. 6

3 APPLICATIONS TO CONTINUOUS- AND FINITE-DEPTH NETWORKS

Noncausal attention achieves universal interpolation with two frozen Gaussian blocks at continuous and finite depth. Causal masking preserves first-token linear relations, ruling out arbitrary interpolation in general while permitting universal interpolation within fixed relation classes.

3.1 NONCAUSAL SELF-ATTENTION NETWORKS

  • For general multihead noncausal attention, 2Nnd frozen Gaussian blocks yield universal interpolation at continuous and finite depth on B_N,n,d.This construction's dictionary size grows with N, unlike the two-block single-head result.Source: §3.1 · pp. 7–8
  • Two frozen Gaussian single-head blocks of width r ≥ d yield universal interpolation for every batch size and sequence length, at continuous and finite depth.The same sampled blocks work simultaneously for all N and n; block choice, signs, and durations depend on the interpolation task.Source: §3.1 · p. 8

3.2 CAUSAL SELF-ATTENTION NETWORKS

  • For causal attention, first-token linear relations persist at every depth, so arbitrary interpolation fails when N > d.Full ambient rank holds exactly when the N first tokens are linearly independent, requiring N ≤ d.Source: §3.2 · pp. 8–9
  • With any prescribed number of heads and positive widths, 2Nnd frozen Gaussian causal blocks yield universal interpolation on each connected component of a fixed first-token-relation class.A single draw works across all components of that class at continuous and finite depth.Source: §3.2 · p. 9

CONCLUSION

The paper establishes universal interpolation for residual self-attention with frozen parameters at continuous and finite depth. It also identifies the trade-off between the number of fixed generators and the depth required for interpolation as a direction for future work.

CONCLUSION

  • Two randomly sampled fixed blocks suffice in the single-head setting, independently of dataset size.The conclusion states universal interpolation for residual self-attention with frozen parameters at both continuous and finite depth.Source: CONCLUSION · p. 9
  • The established universal interpolation result covers both continuous and finite depth.It concerns residual self-attention networks with frozen attention parameters.Source: CONCLUSION · p. 9
  • A future direction is to understand the trade-off between fixed-generator count and interpolation depth.Source: CONCLUSION · p. 10

AI DISCLOSURE SECTION

The authors describe LLM assistance in writing, discovery, research execution, and proof development, and state that they checked all proofs and take responsibility for the paper's final content.

AI DISCLOSURE SECTION

  • LLMs assisted with writing and polishing, including drafting sections that the authors subsequently refined and edited.Source: AI DISCLOSURE SECTION · p. 10
  • LLMs aided retrieval and discovery, code development, research execution, and proof development across several types of mathematical work.Source: AI DISCLOSURE SECTION · p. 10
  • The authors state that all proofs were checked by them and that they take responsibility for the final content.Source: AI DISCLOSURE SECTION · p. 10

A PROOF OF COROLLARY 2.6

The proof uses local full Lie rank to obtain neighborhoods of reachable configurations, then uses connectedness to join any pair by finitely many flows. Conversely, analyticity and exact interpolation imply full Lie rank and a finite field selection at each configuration.

A PROOF OF COROLLARY 2.6

  • Full Lie rank at a point persists on a smaller connected neighborhood, where selected fields and their negatives connect configurations by piecewise-constant controls.Source: A PROOF OF COROLLARY 2.6 · p. 14
  • When M is connected, local reachability and reversible finite flow concatenations imply that any two configurations can be joined by finitely many positive-time flow segments.Source: A PROOF OF COROLLARY 2.6 · p. 14
  • Conversely, for analytic fields, exact interpolation implies full Lie rank at every configuration, with a finite selection of fields sufficient at each point.Source: A PROOF OF COROLLARY 2.6 · p. 14

B PROOF OF COROLLARY 2.7

These proofs connect pointwise spanning and local Lie-rank conditions to exact finite-depth interpolation using fixed parameter dictionaries, including bias-free ResNet results. They also relate analytic full-rank conditions to universal interpolation and identify an obstruction for shared linear dynamics.

B PROOF OF COROLLARY 2.7

  • With fixed parameters chosen before the endpoints, signed lifted updates give universal interpolation; conversely, analytic universal interpolation implies full Lie rank of the fixed lifted fields.The updates are applied on finitely many intervals with a terminal depth S > 0.Source: B PROOF OF COROLLARY 2.7 · p. 14

C JUSTIFICATION OF REMARK 2.8

  • For analytic fields closed under negation, the orbit theorem identifies the Lie algebra’s value with the orbit tangent space, so a single orbit implies full rank everywhere.Negation allows flow segments to be reversed, making the orbit precisely the set reachable by piecewise-constant controls over a finite interval.Source: C JUSTIFICATION OF REMARK 2.8 · p. 14

D POINTWISE FULL-RANK CRITERION

  • A pointwise basis of D locally C1 fields supports exact interpolation of every pair by piecewise-constant controls over some finite interval [0, S].The duration S may depend on the pair.Source: D POINTWISE FULL-RANK CRITERION · pp. 14–15

E.1 DIRECT SPANNING

  • Under pointwise spanning and real scalar closure, every pair in M can be connected exactly by finitely many residual layers, with at most m + D layers.Euler discretization reaches a neighborhood of the target, where a correction using at most D layers sends the endpoint exactly to the target.Source: §E.1 · pp. 15–16

E.2 EXTENSION UNDER A LIE-RANK CONDITION

  • Pointwise Lie rank yields exact finite-depth interpolation between any configurations, with the prescribed problem’s scalar coefficients shared across batch samples.The construction reaches a local correction neighborhood by Euler layers, then appends a fixed finite correction sequence; all intermediate configurations remain in the domain.Source: §E.2 · pp. 17, 19

F.1 DIRECT SPANNING

  • For finite-depth universal interpolation, fixed parameters suffice when each configuration admits Np lifted fields whose values form a basis of RNp.The parameter family allows real scalar multiplication.Source: §F.1 · p. 20
  • In direct spanning, the parameters are fixed before the endpoints; only the depth, indices, and scalar coefficients depend on the interpolation task.The resulting interpolation is at finite depth, with intermediate configurations remaining in MN.Source: §F.1 · p. 20

F.2 EXTENSION UNDER A LIE-RANK CONDITION

  • If finitely many parameters provide locally smooth lifted fields of full Lie rank at every configuration, and real scalar multiplication is allowed, they yield universal interpolation at finite depth using the same parameters.Only the depth, indices, and scalar coefficients vary with the endpoints.Source: §F.2 · p. 20

G RESIDUAL NETWORKS

  • For bias-free ResNets on CN,d with d ≥3, exact and universal interpolation hold at both depths; universal interpolation uses 2d^2 fixed generators independent of N and width one.At continuous depth, each generator may be used in either sign.Source: G RESIDUAL NETWORKS · pp. 20–21

G.1 LINEAR ACTIVATION AND THE OBSTRUCTION TO INTERPOLATION

  • When N > d, shared linear dynamics preserve sample linear relations, so continuous and finite linear residual networks cannot interpolate every pair in any nonempty open configuration set, regardless of width.Arbitrarily close targets in the same connected component can violate a relation preserved by every shared linear map.Source: §G.1 · pp. 21–22

G.2 PROOF OF THEOREM G.1

  • For every d ≥2, the width-one ReLU family exactly interpolates within each connected component of CN,d at both depths; CN,d is connected for d ≥3 but disconnected for d = 2, N ≥2.For d = 2, exact interpolation still holds on each connected component.Source: G RESIDUAL NETWORKS · p. 20, §G.2 · pp. 22–23

G.3 PROOF OF THEOREM G.3

A fixed family of 2d² generators interpolates every pair of configurations in C_N,d at continuous depth, and the same construction gives exact finite-depth interpolation while keeping intermediate configurations in the domain.

G.3 PROOF OF THEOREM G.3

  • The fixed fields realize every matrix in GL+(d) through finite products of elementary exponentials, with subdivision keeping configurations inside C_N,d.Every matrix in GL+(d) is generated by the elementary exponentials; subdivision controls intermediate configurations along the path.Source: §G.3 · pp. 23–24
  • Width-one ReLU flows can be replaced by finite concatenations from the fixed family, enabling the construction to connect arbitrary pairs in C_N,d.The proof conjugates a prescribed flow segment by realizable linear maps, then replaces it with fixed-family flows.Source: §G.3 · p. 24
  • The fixed 2d² generators connect every X and Y in C_N,d by signed-time flows that remain in C_N,d.The generators are fixed before the batch size and endpoints are chosen.Source: §G.3 · p. 24
  • The flows yield exact finite-depth residual layers with the same intermediate configurations, and unused columns embed the blocks into any width r ≥ 1.This construction avoids discretization error.Source: §G.3 · p. 24
  • For N > d, the fixed family’s local Lie-rank condition fails inside an open orthant, so the construction uses activation-pattern changes along its trajectory.Within an open orthant, the fields are locally linear and their lifted Lie algebra has rank at most d² < Nd.Source: §G.3 · p. 24

H PROOF OF LEMMA 3.1

Rank-one noncausal self-attention fields span every token-block direction at each configuration in the generic domain. This pointwise spanning yields exact interpolation on each connected component, continuously and at finite depth.

H PROOF OF LEMMA 3.1

  • At every X in the generic domain, single-head noncausal self-attention fields span every token-block direction.The proof obtains arbitrary vectors in arbitrary target blocks, so the pointwise span is the full configuration space.Source: H PROOF OF LEMMA 3.1 · pp. 24, 26
  • For d ≥ 3, the generic domain is path connected, so exact interpolation holds throughout it at both continuous and finite depth.The theorem also permits any prescribed positive head widths while keeping intermediate configurations in the domain.Source: H PROOF OF LEMMA 3.1 · p. 26
  • The proof uses finite-parameter rank-one fields: limiting softmax arguments establish their span without requiring infinite-temperature layers.Analyticity and scalar rescaling then support the continuous- and finite-depth interpolation arguments.Source: H PROOF OF LEMMA 3.1 · p. 26

I UNIFORM DIRECT SPANNING ON THE GENERIC ATTENTION DOMAIN

A single random bank of 2D attention fields spans the full configuration space simultaneously at every point of the generic domain, almost surely. The proof applies an analytic random-spanning principle to rank-one fields realizable within the architecture.

I UNIFORM DIRECT SPANNING ON THE GENERIC ATTENTION DOMAIN

  • 2D = 2Nnd sampled fields span R^D simultaneously at every generic configuration with probability one under any jointly absolutely continuous draw.This uniform direct-spanning condition requires no Lie brackets.Source: I UNIFORM DIRECT SPANNING ON THE… · p. 27
  • The random-spanning argument bounds the exceptional parameter banks by a measure-zero set when the bank size is at least q + r.A jointly absolutely continuous law therefore avoids banks that fail to span somewhere in the domain almost surely.Source: I UNIFORM DIRECT SPANNING ON THE… · p. 27
  • Rank-one single-head fields are realizable in the prescribed multi-head architecture, and their joint analyticity supplies the random-spanning theorem's regularity condition.Independent Gaussian projection entries have a joint density, so the almost-sure result applies to them.Source: I UNIFORM DIRECT SPANNING ON THE… · p. 28

J PROOF OF THEOREM 3.3

The proof combines uniform direct spanning on the generic domain with a sampled field that separates initially collinear token pairs. Together these properties establish full Lie rank and universal interpolation for the same frozen Gaussian-initialized bank, with terminal depth allowed to depend on the endpoints.

J PROOF OF THEOREM 3.3

  • 2D = 2Nnd Gaussian-initialized noncausal blocks yield universal interpolation at continuous depth on the generic domain with probability one.The same frozen bank works simultaneously for all input and target configurations; block indices, signs, and durations may depend on the task.Source: J PROOF OF THEOREM 3.3 · pp. 28, 32
  • A bank of at least D + 1 sampled fields almost surely provides, at every configuration in B, a field that separates initially collinear pairs.A sufficiently small nonzero residual step with that field moves the configuration into A.Source: J PROOF OF THEOREM 3.3 · pp. 28, 30
  • After moving into A, uniform direct spanning and invertible flow pullbacks give full Lie rank throughout B for the same bank.The argument applies to independent Gaussian entries with any prescribed positive variances, or more generally to jointly absolutely continuous draws.Source: J PROOF OF THEOREM 3.3 · p. 32
  • The terminal horizon may depend on the endpoints; a fixed horizon cannot reach every admissible rescaling of an input.The proof gives a growth bound for trajectories and notes that arbitrarily large positive rescalings remain in the domain.Source: J PROOF OF THEOREM 3.3 · p. 32

K PROOF OF COROLLARY 3.4

This proof section establishes finite-depth interpolation using a bank fixed before endpoints are chosen, alongside a two-field interpolation result on a domain that may depend on sampled parameters. It develops the supporting Lie-rank, polynomial-lift, and exponential-independence arguments to rule out common invariant hypersurfaces for the two fields.

K PROOF OF COROLLARY 3.4

  • A fixed bank yields finite-depth interpolation on B_N,n,d by placing at most two residual layers around interpolation on A_N,n,d; indices and gains are shared across the batch.The bank is chosen before endpoints, and the final layer is an admissible forward residual map.Source: K PROOF OF COROLLARY 3.4 · pp. 32–33

L PROOF OF THEOREM 3.5

  • Two frozen base fields, with either sign over freely chosen intervals, give universal interpolation on a full-measure, path-connected domain that may depend on the sampled pair, N, and n.The matrices are independent of N and n; the result does not assert full Lie rank on all of B_N,n,d or a width-one realization.Source: L PROOF OF THEOREM 3.5 · p. 34

L.1 A FINITE LIE-MINOR WITNESS FOR EVERY BATCH SIZE

  • For d ≥ 3, a finite Lie-minor witness yields an open, dense, full-measure full-rank locus in B_N,n,d simultaneously for every n ≥ 2 and N ≥ 1 under a generic parameter draw.The same two generators work for every pair (n, N), by analyticity of the determinant and a countable intersection of probability-one events.Source: §L.1 · pp. 34–35

L.2 GENERIC MATRIX CONDITIONS

  • The additional matrix conditions hold almost surely independently of N and n, including Lie generation by V1, V2, nonproportional matrix components, and simple spectra without eigenvalue pairs summing to zero.These conditions are imposed alongside the generic-rank event.Source: §L.2 · p. 36

L.3 EXCLUSION OF INVARIANT ALGEBRAIC HYPERSURFACES

  • On a connected regular real hypersurface patch, an identity among exponentials of polynomial exponents with pairwise nonconstant differences forces every polynomial coefficient to vanish on the patch.The conclusion concerns coefficient restrictions to the patch, not their values outside it.Source: §L.3 · p. 36
  • Under conditions 1, 2, and 3 of Section L.2, no nonconstant real polynomial hypersurface patch is tangent to both batch attention fields.The patch is a nonempty connected relatively open subset of the hypersurface.Source: §L.3 · p. 38
  • The proof rules out two nonresonant generators by generating all tangent directions, then excludes two resonant generators and the remaining single-resonant case using conditions 2 and 3.At least one generator must be resonant; two resonant generators contradict condition 2, while the remaining case contradicts condition 3.Source: §L.3 · pp. 40–41

L.4 POLYNOMIAL LIFTS AND COMMON INVARIANT DIVISORS

  • With linearly independent score quadratics and invertible LV1, LV2 on homogeneous quadratics, the polynomial-lift argument identifies a unique largest-weight monomial exponent β* across admissible weight choices.The lifted attention fields yield polynomial derivations; coefficient constraints force each monomial of Br to contain one score variable per token position and have state-coordinate degree at most two.Source: §L.4 · pp. 42–43
  • Varying either score block’s weights freely forces every monomial exponent in P to coincide, so P = z^βp(X).The argument first passes from weights without ties to all strictly admissible weights, then varies the other block without bound to force equality of its exponent coordinates.Source: §L.4 · p. 43

L.5 ANALYTIC HYPERSURFACES AND THE AX–SCHANUEL THEOREM

  • A smooth real-analytic hypersurface tangent to both fields yields a nonunit irreducible P in A invariant under both cleared derivations.The proof complexifies a local parametrization; its image is Zariski dense, and tangency identities extend to the complex parameter domain.Source: §L.5 · p. 44
  • Ax–Schanuel gives transcendence degree at least p + D − 1 for the functions X(t) and e^Q(X(t)).Applied to the p score quadratics and D coordinates, it first gives at least p + 2D − 1 after adjoining coordinate exponentials; removing those auxiliary exponentials lowers the bound by at most D.Source: §L.5 · p. 44
  • The Lie determinant supplies a relation, making the generated field’s transcendence degree exactly D + p − 1 and the relation ideal principal, I = (P).Tangency then gives δ_rP = B_rP for r = 1, 2, so P is invariant under both cleared derivations.Source: §L.5 · p. 44

L.6 CONNECTEDNESS OF THE INTERPOLATION DOMAIN

With probability one, the same two frozen Gaussian attention blocks provide continuous-depth universal interpolation on a connected, full-measure domain for every finite batch and sequence size. The result also yields finite-depth interpolation on that domain, with endpoint-dependent block choices, gains, and depth.

L.6 CONNECTEDNESS OF THE INTERPOLATION DOMAIN

  • With probability one, two frozen noncausal single-head Gaussian blocks of width r ≥ d yield continuous-depth universal interpolation on a connected, full-measure domain for every finite N and n.The result assumes d ≥ 3 and n ≥ 2; the same sampled pair works simultaneously for all such N and n.Source: §L.6 · pp. 45–46
  • The exceptional set has dimension at most D − 2, leaving a connected, open domain whose complement has measure zero.The rank test defines the exceptional set analytically; the excluded configurations also include proportional token pairs, zero tokens, and permutation/sign-related sequences.Source: §L.6 · p. 46

L.7 EXACT INTERPOLATION AT FINITE DEPTH

  • On that same domain, finite-depth interpolation uses the same two frozen fields with shared scalar gains; depth, block indices, and gains may depend on the endpoints.The argument is deterministic for each N and n once the probability-one draw is fixed.Source: §L.7 · p. 47
  • The two-block conclusion is restricted to the connected generic domain, and no bound on the required depth is asserted.The exceptional set may be nonempty; Theorem 3.3 instead uses 2Nnd blocks for interpolation throughout BN,n,d.Source: §L.7 · p. 47

M NUMERICAL TESTS OF THE HEAD-WIDTH THRESHOLD

Numerical tests support the necessary head-width threshold mr ≥ d, while results at the threshold are sensitive to conditioning and do not establish universal interpolation. The experiments cover finite configurations and show full-rank counts varying with projection variance and tolerance.

M NUMERICAL TESTS OF THE HEAD-WIDTH THRESHOLD

N PROOF OF PROPOSITION 3.8

Causal masking makes first-token updates linear, preserving every initial linear relation among batch first tokens. Consequently, when N > d, causal interpolation cannot reach arbitrary targets; later-token directions remain unrestricted.

N PROOF OF PROPOSITION 3.8

  • Every causal field and iterated Lie bracket has first-token block AX1 for a constant matrix A, restricting first-token motion to that linear image.For two fields with first-token blocks AX1 and CX1, the bracket has first-token block (CA − AC)X1.Source: N PROOF OF PROPOSITION 3.8 · pp. 48–49
  • All later-token directions can be varied independently without changing AX1, so the first-token constraint is the remaining rank restriction.The first-token image has dimension d rank X1.Source: N PROOF OF PROPOSITION 3.8 · p. 49
  • When N > d, a first-token relation can be broken by a target in the same component, but causal flows and finite causal residual networks preserve the original relation and cannot reach it.The obstruction follows from the first-token update being a common linear map applied to the batch first tokens.Source: N PROOF OF PROPOSITION 3.8 · p. 49

O PROOF OF COROLLARY 3.9

For batches sharing a fixed first-token kernel, causal fields preserve the associated linear space and achieve universal interpolation on each connected component. The conclusion transfers to finite residual networks as well as continuous depth.

O PROOF OF COROLLARY 3.9

  • For a fixed first-token kernel K, the admissible configurations form a relatively open subset of a linear space, so each connected component is relatively open.The linear space has dimension N(n−1)d + dr, where r = N − dim K.Source: O PROOF OF COROLLARY 3.9 · p. 49
  • Causal fields preserve the linear space associated with K, and every configuration in it satisfies the first-token rank condition of Proposition 3.8.The first-token block of any such configuration can be written Z1 = AX1 for some linear map A.Source: O PROOF OF COROLLARY 3.9 · p. 49
  • On each connected component, interpolation follows at continuous and finite depth, with constructions staying in that component.The rank-at-most-one fields can be realized with one head of width one; additional prescribed heads can be set to zero.Source: O PROOF OF COROLLARY 3.9 · p. 49

P PROOF OF COROLLARY 3.10

For a fixed K, the proof uses Gaussian-factor samples to obtain a signed family of generators with full rank throughout the configuration class. It then derives exact interpolation within connected components and transfers it to finite depth.

P PROOF OF COROLLARY 3.10

  • 2R independent Gaussian-factor samples span simultaneously throughout the configuration domain with probability one, for the fixed K.The proof applies a theorem with both configuration and output dimensions equal to R.Source: P PROOF OF COROLLARY 3.10 · pp. 49–50
  • The signed family has pointwise full rank, yielding exact interpolation within each connected component over a horizon that may depend on the endpoints.The same sampled generators and their signs give the relative universal interpolation property.Source: P PROOF OF COROLLARY 3.10 · p. 50
  • A sufficiently fine Euler discretization gives finite-depth exact interpolation, with at most R correction layers and total depth at most L0 + R.The sampled matrices are fixed before the endpoints are chosen, and scalar gains are shared across the batch.Source: P PROOF OF COROLLARY 3.10 · p. 50
Loading 2610.01981v1…