PaperlayerView exact revision on arXiv

Source-linked AI summary

Universality and Generalization of Causal Transformers Across Context Lengths

Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré

arXiv:2609.35055v1cs.LGmath.OCstat.ML

TL;DR

Most universality results choose a different network for each fixed sequence length, leaving open whether one causal transformer can approximate rules across resolutions. This paper characterizes such approximation under shared input regularity and cross-resolution continuity, then extends it to continuous paths and learning guarantees. Its regression bound has no maximum-length factor, while empirical support for Hölder regularity is confined to observed scales and varies by dataset.

  • Problem

    Most expressivity results fix sequence length before choosing a network, whereas long-context systems apply one learned rule across lengths.

    Source: §1 · p. 1

  • Method

    The paper combines a common input modulus with asymptotic cross-resolution compatibility to characterize uniform approximation by one causal transformer and its continuous-time limit.

    Source: §1 · pp. 1–2

  • Results

    Bounded-weight empirical risk minimization achieves root mean-square prediction error O((log log N/log N)^(β/(d+2))) at fixed confidence, without a maximum-length factor.

    Source: §1 · p. 2, §4 · p. 11

  • Takeaways & Limitations

    Continuously extendable causal families are exactly those uniformly approximable across resolutions by a length-independent masked transformer, which also approximates the continuous path limit.

    Source: §4 · p. 11

    Empirical regularity is a finite-scale diagnostic: fitted slopes do not certify an asymptotic uniform exponent, and the ETT channels share one source.

    Source: §3 · pp. 9–10

Abstract

from arXiv · show

Long contexts are central to modern transformer systems, but most expressivity results choose a different network for each fixed sequence length. We study whether one masked transformer can approximate causal token-to-token maps uniformly over sequences of arbitrary length sampling a fixed normalized horizon. To relate sampling resolutions, we model tokens by $\alpha$-H\"older sequences or, more generally, a common modulus of continuity. Our notion of continuity across resolutions characterizes the causal families admitting uniform approximation on these compact input classes by a single transformer with length-independent parameters. The result extends to the infinite-length mean-field limit, where tokens form continuous curves and masked attention becomes a causal time integral. For bounded regression with target maps satisfying a $\beta$-smooth stability condition defined using regular test functions, quantitative approximation yields a generalization bound: exact empirical risk minimization over suitably sized bounded-weight transformers gives root mean-square prediction error $O((\log\log N/\log N)^{\beta/(d+2)})$ from $N$ iid labeled sequences. The bound holds at fixed confidence on the same sampling distribution, with $d$ the token dimension and no maximum-length factor. Finally, experiments on physical time series support the H\"older-regular token model at observed scales, with dataset-dependent fitted exponents, whereas text input embeddings provide a contrasting case. Native and dense sampling, shuffled controls, and refinement checks delimit this empirical regularity regime.

1 Introduction

The paper asks whether one causal transformer can approximate token-to-token rules uniformly across resolutions, rather than requiring a different network for each fixed length. It connects this approximation to a continuous-time limit, learning guarantees under additional regularity, and empirical tests of token geometry.

Transformers and long-context architectures

  • The paper’s all-resolution setting means increasingly fine samples of a fixed normalized horizon under common regularity, not unrestricted growth of a symbolic string.Prior transformer results generally fix sequence length before selecting the network; this work instead asks about one parameter set across resolutions.Source: §1 · pp. 1–2

Contributions and organization

  • The same finite architecture yields temporal-attention universality in the continuous-path limit, where causal attention arises as the refinement limit of finite grids.The continuous-time result is distinct from a continuous-depth limit.Source: §1 · p. 2
  • One shallow masked transformer approximates continuously extendable causal target families uniformly across sequence lengths, with parameters independent of length.The target may depend on sequence length, but must converge across refinements to a continuous path operator.Source: §1 · p. 2
  • The resulting learning guarantee gives root mean-square prediction error O((log log N/log N)^(β/(d+2))) at fixed confidence from N iid variable-length sequences.It uses clipped exact empirical risk minimization over suitably sized bounded-weight transformers; the capacity estimate is independent of context length.Source: §1 · p. 2
  • Physical-signal and continuous pretrained-feature increments show dataset-dependent finite-scale decay, while text input embeddings provide a contrasting case.Native and dense sampling, shuffled controls, and refinement checks delimit the empirical support for the Hölder input model.Source: §1 · p. 3

2 All-resolution causal universality

This section studies approximation of compatible causal target families across context lengths by a single length-independent transformer, including discrete and continuous-time settings. It also gives a fixed-confidence root mean-square generalization rate for exact ERM on variable-length sequences, without a maximum-length or sequence-length moment requirement.

2.1 Discrete causal transformer

  • The discrete architecture uses the same dimensions and parameters at every length, with normalized position retained in the hidden state to provide resolution information.A fixed finite causal transformer induces a causal, continuously extendable family across resolutions; compatibility is asymptotic and permits arbitrary causal changes at finitely many lengths.Source: §2.1 · pp. 3–4, §2 · p. 3, §2.2 · p. 6

2.2 Discrete causal targets with a continuous path extension

  • A common input modulus makes the token grids part of one compact path class, while target maps must be causal at each length and converge uniformly to a continuous path operator.The extension preserves causality and is unique; controlled ODE targets provide an example with extension error at most M/n.Source: §2.2 · pp. 5–6

2.3 Universality theorems

  • For every accuracy, one fixed transformer approximates a continuously extendable causal family uniformly across all sequence lengths; continuous extendability is also necessary for uniform all-resolution approximation.The continuous-time architecture likewise approximates continuous causal path operators, and its temporal attention is the refinement limit of discrete attention.Source: §2.3 · pp. 6–7

2.4 Quantitative rates and generalization bound

  • For β-smooth targets, the uniform discrete–path approximation error is O(Rβ;ω(log log p / log p)^(β/(d+2))) with p parameters.The regularity condition is Hölder stability for 0 < β ≤ 1 and stability through smooth token–time averages with Lipschitz endpoint dependence for β > 1; the section specifies squared-loss ERM but provides no generalization-bound result in these passages.Source: §2.4 · p. 8
  • The variable-length-sequence guarantee covers teachers in Fβ(ω) for an admissible common modulus, with constants independent of the sampling distribution and context lengths.No upper bound or moment assumption on sequence length is needed.Source: §2.4 · p. 9
  • The proof combines comparator approximation with estimation using logarithmic-size covers independent of context length and Bernstein concentration; a nonzero optimization gap adds 2τopt.The clipped class has a uniform cover of size O(p log(p/ϵ)).Source: §2.4 · p. 9
  • O((log log N/log N)^{β/(d+2)}) is the exact-ERM root mean-square rate at fixed confidence, with N counting independent sequences whose tokens may be dependent.The guarantee is for sufficiently large N.Source: §2.4 · p. 9

3 Empirical diagnostics of Hölder path geometry

The experiments test finite-scale increment regularity across physical signals and representations, finding positive but representation- and dataset-dependent patterns that do not establish a common Hölder bound at all resolutions.

Physical signals and continuous content

  • Physical recordings support a positive but representation-dependent finite-scale regularity regime, not one common Hölder ball across all resolutions.Raw physical-signal mean slopes span 0.041–0.389; fitted slopes are finite-scale diagnostics rather than certified asymptotic exponents.Source: §3 · pp. 9–10

A common scale diagnostic

  • The diagnostic fits worst increments over dyadic lags and use a finer band with at least four scales per primary fit.The maximum over positions probes increments relevant to a uniform Hölder condition; a fitted slope alone does not establish that bound.Source: §3 · p. 9

Figure 1

  • Figure 1 normalizes each window by its finest-lag increment, preserving its fitted slope while fixing the first plotted point at one with zero SD.Curves use normalized lag; shading is one sample SD, and dashed lines mark fit cutoffs.Source: Figure 1 · p. 10

Physical signals and continuous content

  • All 29 dense Bolt windows have positive slopes, largely lost after shuffling; weather and oil temperature retain positive native-stride slopes, while appliances and ETT load are weaker.The same frozen frontend is applied to identical 16-sample patches at strides 1 and 16; native outputs match corresponding dense outputs exactly.Source: §3 · p. 10
  • Delay patches often have larger slopes, so learning alone does not explain the observed geometry.Overlap, scale selection, jumps, and mixed nested-grid seminorm trends qualify these findings; the ETT channels share one source.Source: §3 · p. 10

Text as a contrasting input model

  • At N = 16,384, BigBird text input embeddings have nearly flat worst-increment curves, with mean slopes 0.011–0.013 across seven sources.Appending words is not refinement of a fixed signal; these finite-scale results neither establish asymptotic failure nor rule out other representations.Source: §3 · p. 10

Table 1

  • The native, dense, and shuffled comparison reports fine-band slopes for physical Bolt content and WikiText-103 BigBird input; native dashes indicate insufficient scales.The text shuffled control is unmeasured.Source: Table 1 · p. 10

4 Conclusion

The paper characterizes uniform all-resolution causal approximation by length-independent masked transformers and connects it to continuous path limits. It also derives learning guarantees under target regularity and reports empirical regularity diagnostics whose scope remains finite-scale.

Conclusion

  • Continuously extendable causal families are exactly those uniformly approximable across resolutions by a length-independent masked transformer, which also approximates the continuous path limit.The paper derives logarithmic prediction rates under β-smooth target stability; matching minimax exponents remain open.Source: §4 · p. 11
  • Bounded-weight empirical risk minimization gives logarithmic prediction rates from independent labeled sequences without a maximum-length factor.The conclusion identifies geometry-adaptive rates, efficient width–depth tradeoffs, dependent samples, and computationally feasible optimization as future work.Source: §4 · p. 11
  • Physical pretrained features show dataset-dependent finite-scale regularity, while text input embeddings have nearly flat worst increments.Sampling and jump controls qualify these diagnostics, which do not certify a common all-resolution Hölder bound.Source: §4 · p. 11

A Positional encoding is needed for universality

Normalized position in the hidden state is needed for the stated all-resolution universality result: masking and score-side position alone cannot recover the normalized clock on constant inputs. The obstruction persists even when architectures vary by resolution.

Positional encoding is needed for universality

  • The architecture allows attention weights to depend on length, positions, and causal hidden prefixes, but its values and pointwise maps are shared and position-independent.Its initialization depends on total length but not token index; it has no distinguished start token or position-dependent value input.Source: A Positional encoding is needed for… · p. 17
  • Score-only positional normalized transformers cannot recover the normalized clock on constant inputs, even when architecture and parameters vary with resolution.The lower bound applies to every admissible common input modulus.Source: A Positional encoding is needed for… · p. 17
  • The obstruction is constant-input invariance: unit-mass normalized attention maps identical values to identical values, and shared pointwise layers preserve that equality.This holds even with signed, position-dependent attention weights.Source: A Positional encoding is needed for… · p. 18
  • For the normalized clock family, every all-resolution architecture in this class has error at least 1/2; the all-resolution bound is sharp.The paper states that total length and score-side position cannot replace a position-revealing operation outside the normalized score channel.Source: A Positional encoding is needed for… · pp. 17–18

A.1 Proof

The proof reduces the positional obstruction to invariance of constant hidden states under normalized aggregation. The quantitative-rate framework measures target stability on causal histories using smooth-test distances, with regularity assumptions and scope that differ between low and high orders.

A.1 Proof

  • A constant hidden sequence remains constant after normalized attention, and shared pointwise maps preserve equality across positions.The temporal counterparts preserve paths that are constant in time.Source: §A.1 · p. 18
  • Comparing first and last clock values gives the fixed-resolution bound, while a shared constant predictor attains the sharp all-resolution error 1/2.The fixed-length constant predictor attains error (1−1/n)/2; the shared predictor is 1/2 in discrete and continuous time.Source: §A.1 · p. 19

B Quantitative approximation rates

  • The rate analysis first treats an arbitrary admissible input modulus and transfers a discrete-resolution estimate to continuous time before specializing to Hölder histories.Its smooth-test distances measure variation between histories that finite attention probes cannot yet distinguish.Source: B Quantitative approximation rates · p. 20, §B.1 · p. 20

B.1 β-smooth target regularity

  • The endpoint term retains the current token and its position exactly, which is essential for the rate because a positive-time prefix assigns zero mass to its terminal token.Attention writes only to auxiliary coordinates, leaving the endpoint coordinates carried by the residual connection to the readout.Source: §B.1 · p. 20
  • β-smooth regularity is Lipschitz stability of the common readout on causal histories under the paper’s augmented smooth-test metric.For 0 < β ≤ 1 it is equivalent to ordinary β-Hölder continuity in the causal-history metric; for β > 1, smooth test functions measure higher-order regularity with a first-order endpoint term.Source: §B.1 · p. 22
  • The smoothness order β is fixed as the parameter budget grows, constants may depend nonuniformly on β, and the paper makes no analytic-class claim.The regularity seminorm and learned readout may also depend on the input modulus ω.Source: §B.1 · p. 22

B.2 Quantitative all-resolution rate

For β-smooth causal families, the section gives a shared-transformer approximation guarantee across finite resolutions and the continuous-time boundary. It explains the probe-based complexity and the scope of the smoothness condition.

B.2 Quantitative all-resolution rate

  • A shared transformer approximates continuously extendable, β-smooth causal families jointly across sequence lengths and their continuous-time realization.Its dimensions, matrices, and biases are independent of sequence length.Source: §B.2 · p. 23
  • The discrete approximation error transfers to the continuous boundary without additional heads, parameters, or approximation loss.Source: §B.2 · p. 24
  • On a normalized target ball, parameters can have finite dyadic descriptions, but the guarantee is normalized absolute error rather than error scaled by the smoothness seminorm.The architecture is fixed across that ball at each budget; evaluation is analyzed in exact arithmetic.Source: §B.2 · pp. 24–25

B.3 Proof mechanism: quantitative resolution by causal probes

  • Causal attention probes recover moments and control history metrics; β-regularity then bounds target oscillation on feature cells before a shallow ReLU readout is fitted.Counting the probes and readout yields the all-resolution parameter rate.Source: §B.3 · p. 25

B.2 Quantitative all-resolution rate

  • log par(TK)=Oβ(K^(d+2) log(K+1)); the exponent d+2 comes from feature dimension and stability, not the input modulus.Source: §B.2 · p. 24, §B.3 · p. 25

B.4 Interpreting the smoothness condition

  • Lipschitz endpoint-only maps satisfy the smoothness condition at every fixed β>1 without being classically differentiable; retaining endpoint coordinates adds quantitative stability beyond the law’s exact information.Source: §B.4 · p. 26
  • A sufficient high-order criterion uses an ambient extension with a uniformly Cβ functional derivative and Lipschitz control in the retained endpoint; behavior away from the history graph is auxiliary.Source: §B.4 · p. 26
  • Lipschitz readouts of smooth history statistics are β-smooth without requiring differentiation of their outer map.The integral terms are controlled by smooth-test norms, with the endpoint controlled separately.Source: §B.4 · p. 26

B.5 Logarithmic lower bounds on Hölder histories

This section develops a logarithmically slow worst-case approximation lower bound for finite-description models on Hölder histories, and explains why it does not directly extend to unrestricted real-parameter transformers. Its proof connects smooth first variation to target stability and gives codebook, causal-probe, and transformer-construction bounds that are uniform across input moduli and prefix lengths.

B.5 Logarithmic lower bounds on Hölder histories

  • The finite-code lower and constructive upper bounds are both logarithmic but have exponents β and β/(d + 2), leaving the dimension penalty and unrestricted real-parameter optimality unresolved.The lower bound does not establish optimality of the upper exponent, and its constants may depend on α and L unlike the normalized upper-bound constant.Source: §B.5 · pp. 27–28
  • Finite-description worst-case error decays no faster than (log B)^-β for α = 1 and (log log B/log B)^β for each fixed α < 1.A multilevel packing of Hölder paths creates separated prefix laws; independently signed target values on the packing yield more admissible targets than a B-bit description can distinguish.Source: §B.5 · p. 27
  • Fixed-size shallow transformers have infinite threshold capacity on Hölder histories, even with weights in [−1, 1], so VC dimension cannot yield the finite-description lower bound for real parameters.A single temporal head can threshold Laplace moments using a real-valued parameter; shattering may use increasing common sequence lengths, and margins can vanish, while bounded-weight predictors still admit finite uniform covers at positive accuracy.Source: §B.5 · p. 28

C.1 A finite family of exact masked-attention probes

  • A finite family of exact causal masked-attention probes resolves histories in smooth-test and Wasserstein metrics, with feature scales shared across fixed smoothness exponents.The proof chain uses exact probes to recover moments, polynomial approximation to control smooth tests, and a counted shallow readout; feature closeness also controls endpoint coordinates.Source: C Proofs for quantitative rates and… · p. 29, §C.1 · pp. 32–33

C.2 A first-variation certificate for high-order regularity

  • For β > 1, a smooth first-variation condition yields the required β-smooth target stability by combining regularity in the history law with first-order endpoint stability.The proof requires the relevant estimates only along mixture segments between admissible histories and at the endpoint states in the split.Source: §C.2 · pp. 33–34

C.3 A dimension-explicit shallow ReLU readout

  • A dimension-explicit lemma constructs a one-hidden-layer ReLU readout for Lipschitz maps, using extension, mollification, and ridge approximation.The argument uses a McShane extension clipped to the target range, followed by a counted ridge-network approximation.Source: §C.3 · pp. 34–35

C.4 Quantitative readout and total parameter count

  • Feature-cell target oscillation is O(K^-β), while log(2 + W_K) ≤ Cβ K^(d+2) log(K + 1), with d + 2 arising from joint token–time features and their stability cost.The feature-resolution step obtains the full K^-β oscillation before the Lipschitz envelope and shallow ReLU approximation.Source: §C.4 · pp. 36–37

C.5 Finite precision on the normalized target ball

  • On the normalized target ball, a common scale-K architecture has bounded coefficients and output-row norms, and perturbing each parameter by η ≤ 1/p controls attention-state changes uniformly.The construction tracks parameter storage separately from exact network evaluation, with bounds on the dense parameter count, coefficient magnitudes, and output-row norms at most p.Source: §C.5 · pp. 39–40
  • The perturbation estimate covers every discrete prefix and normalized prefix integral for t > 0 without prefix-length dependence, and the head average at t = 0 is bounded directly.Source: §C.5 · p. 40
  • A dyadic description of the dense parameters has length at most C_β p log^2 p bits, yielding the stated budget-inversion bound.The common header costs O_{d,d′,β}(log p) additional bits; fixed-length descriptions use the same field length for all dense parameters, including zeros.Source: §C.5 · p. 41
  • For sufficiently large B, choosing p = ⌊c_β B / log^2 B⌋ produces a B-bit codebook of at most 2^B elements approximating the whole normalized target ball.The finite-code radius follows because log p ≍ log B and log log p ≍ log log B.Source: §C.5 · p. 41
  • The architecture, coefficient, precision, and error bounds are independent of the admissible input modulus, although selected coefficients may depend on the input class and target.Source: §C.5 · p. 41

C.6 Proofs of the history-complexity and capacity bounds

The capacity proofs build many distinguishable Hölder histories and use them to establish finite-description lower bounds and infinite VC dimension. The accompanying theorem proofs assemble a common finite and continuous-time approximation argument.

C.6 Proofs of the history-complexity and capacity bounds

  • A greedy packing of multilevel histories yields a finite-description lower bound for completed causal families.The construction separates histories using smooth tests and transfers the separation to target families.Source: §C.6 · pp. 42–43
  • A one-head, one-hidden-unit ReLU subclass has infinite VC dimension, while its parameter count stays fixed at 5(d + 3) + 1 + 2d′.The proof fixes histories before selecting labels.Source: §C.6 · p. 44
  • For every N, sufficiently fine sampled sequences preserve the threshold labelings and use unchanged heads, readout weights, and parameter count.Right-endpoint Riemann sums converge simultaneously for the finite collections of histories and labelings.Source: §C.6 · p. 44

D Proof of Theorems 2.4 and 2.6

  • The proof of Theorems 2.4 and 2.6 proceeds through compactification, target compatibility, prefix-law identification, architectural closure, Laplace separation, one-block realization, and Stone–Weierstrass assembly.The closure argument applies to fixed finite-depth transformers; an appendix upgrades convergence from step to affine interpolants.Source: D Proof of Theorems 2.4 and 2.6 · pp. 44–45

D.1 Detailed proof reading guide

The proof guide reduces approximation to a continuous function of causal-history states, then constructs separating attention probes and realizes their approximations with one block and a shared readout.

D.1 Detailed proof reading guide

  • One compact evaluation space combines finite token predictions with continuous-path predictions, while its resolution coordinate tracks finite grids and their path boundary.Common-modulus inputs make this state space compact, and finite states are dense.Source: §D.1 · p. 45
  • Normalized token–position prefix laws identify causal-information classes, yielding a continuous factor from the compact history space.The laws retain the observed prefix while forgetting the future.Source: §D.1 · p. 45
  • A scalar masked-attention head computes separating probes exactly, and equality of all probes identifies the histories.The probes identify Laplace transforms, then polynomial moments and the laws.Source: §D.1 · p. 46
  • Stone–Weierstrass approximates the continuous factor with a vector polynomial, which one attention block and one shared pointwise ReLU network implement with parameters independent of sequence length.The same parameters establish the finite and boundary results.Source: §D.1 · p. 46

D.2 The joint compactification

The proof places finite-resolution predictions and continuous-path predictions in one compact state space. It then establishes target compatibility across resolutions and represents causal histories by prefix laws.

D.2 The joint compactification

  • A compact evaluation space records the interpolated path, prediction time, and resolution, with finite states dense at the path boundary.Sampling and reconstruction preserve the common modulus and approximate paths uniformly as grid spacing shrinks.Source: §D.2 · pp. 47–48

D.3 Compatible targets and inherited boundary causality

  • For continuously extendable causal families, joint continuity, uniform path extension, and the stated uniform extension condition characterize compatible targets; finite prefix causality passes to the boundary.The continuous path extension is unique when the uniform extension condition holds.Source: §D.3 · pp. 48–49

D.4 Prefix laws identify causal histories

  • The continuous prefix-law map identifies states exactly when their observed finite prefixes or boundary path prefixes agree, giving a compact representation of causal histories.The quotient by prefix equivalence is homeomorphic to the space of prefix laws, and continuous targets factor through that representation.Source: §D.4 · pp. 50, 52

D.5 Discrete attention closes under fixed-depth compositions

The section establishes that masked attention remains uniformly extendable across resolutions through fixed-depth compositions, and constructs approximations using shared attention and ReLU components. One construction gives both theorem estimates with the same parameters when the continuous target extends the discrete family, while qualitative universality also holds with bounded matrices for each scalar attention head.

D.5 Discrete attention closes under fixed-depth compositions

  • Uniform convergence of masked attention to temporal attention survives every fixed composition, so each fixed transformer extends to a continuous causal family across resolutions.The extension uses temporal attention as its continuous-path limit.Source: §D.5 · pp. 52–53

D.6 Log-Laplace probes separate prefix laws

  • Equality of all Log-Laplace probes La,c for unit-ball directions and c in [0, 1] implies equality of the compactly supported prefix laws.The probes are continuous, and their values determine the law.Source: §D.6 · pp. 54–55

D.7 Parallel realization by one masked attention layer

  • One shared shallow masked-attention block computes finitely many Log-Laplace probes in parallel, while sequential probe-polynomial realization keeps width and head count independent of sequence length.The shallow construction uses a ReLU readout; the sequential construction permits finite depth and at most d′ scalar heads per attention block.Source: §D.7 · pp. 55–56

D.8 Stone–Weierstrass assembly

  • Because Log-Laplace probes separate points of the compact prefix-law space, their generated algebra is dense in continuous functions, enabling finite probe-polynomial approximations realized by a shared attention block and ReLU readout.The cited construction approximates the polynomial simultaneously on every finite state and boundary state.Source: §D.8 · p. 58
  • The construction yields the finite-state and boundary-state theorem estimates, with one parameter list when the continuous target extends the discrete family.The passage identifies these restrictions as the two theorem claims.Source: §D.8 · p. 59
  • Qualitative universality holds with all four matrices of each scalar attention head bounded, and the same bounds apply to the serialized realization.The query is exactly one and the key depends only on y in Ω × [0, 1].Source: §D.8 · p. 59
  • Universality does not require increasingly concentrated attention, while head count, serialized depth, widths, and parameter magnitudes remain unrestricted.The bounds are per head, not a bounded-weight guarantee for the whole network.Source: §D.8 · p. 59

D.9 The exact all-resolution approximation class

The all-resolution approximation class is exactly the continuously extendable causal families, and controlled ODEs provide examples of such families and their refinement behavior.

D.9 The exact all-resolution approximation class

  • Continuously extendable causal families are exactly the uniform all-resolution closure of shallow and finite-depth masked transformers.Increasing depth does not enlarge this closure.Source: §D.9 · pp. 59–60
  • Even-length and odd-length terminal outputs of the alternating-sign family force uniform approximation error of at least 1.The obstruction is its cross-resolution discontinuity, not the inability to distinguish a particular pair of lengths.Source: §D.9 · p. 60

E.1 Construction and uniform compatibility

  • Controlled ODE solutions define a continuously extendable causal family with uniform state and speed bounds independent of input and resolution.Every solution exists uniquely on [0, 1] and is M-Lipschitz in time.Source: §E.1 · p. 61

E.2 Refinement, an explicit instance, and scope

  • For the controlled integrator, affine-control outputs differ from right-endpoint cumulative sums by at most 1/n, yet share the same continuous path limit.The compatibility defect is at most 1/n.Source: §E.2 · p. 63
  • The ODE example establishes qualitative extendability, but quantitative learning rates additionally require finite β-smooth target regularity.Continuity of the vector field in its control argument alone does not ensure that condition.Source: §E.2 · p. 63

E.3 β-smooth controlled integrators

The β-smooth controlled-integrator example shows that discretization affects target regularity, while temporal attention replaces masked averages with causal integrals over ordered histories.

E.3 β-smooth controlled integrators

  • The right-endpoint ODE teacher is continuously extendable, and centering and scaling place its family in the normalized β-smooth class.Its optimal seminorm may vary with the common modulus, while the displayed upper bound is uniform over the domains.Source: §E.3 · p. 63
  • For β > 1, the exact affine-control integrator fails the β-smooth condition, whereas the right-endpoint family satisfies it despite sharing the same path limit.For the tested pair, the right-endpoint outputs agree; the exact affine-control outputs do not.Source: §E.3 · p. 64

F.1 Temporal attention and the continuous-time realization

  • Under uniform convergence of hidden-state interpolants, masked softmax averages converge to normalized temporal integrals over observed history.Temporal attention integrates the ordered trajectory only over [0, t], retaining temporal order rather than averaging over the full token law.Source: §F.1 · pp. 64–65

F.2 Path-space universality

Temporal attention is universal on compact path classes controlled by a common modulus, and fixed causal transformers converge to their temporal realizations under grid refinement.

F.2 Path-space universality

  • Temporal attention is universal for continuous causal targets on compact path classes controlled by an admissible common modulus.The theorem is stated for path targets without finite-grid targets in its assumptions or conclusion.Source: §F.2 · p. 65

F.3 Uniform closure under grid refinement

  • A fixed finite-depth causal transformer converges uniformly to its temporal realization when grid inputs converge uniformly to a path.The parameter list is fixed independently of sequence length.Source: §F.3 · p. 65

G Proof of the generalization bound

This section develops the generalization-bound analysis, including its approximation, parameter-cover, concentration, and training considerations. It also describes the empirical regularity estimator and clarifies that finite-band fits do not establish theorem-level Hölder regularity.

G.3 Squared loss on independent complete sequences

  • Concentration treats each complete sequence as one bounded-loss observation: independence is required across the N sequences, not among tokens within a sequence.Conditional centering identifies excess squared loss with prediction error, and Bernstein’s inequality gives a finite-class fast oracle inequality.Source: G Proof of the generalization bound · p. 66, §G.3 · pp. 69–70

G.1 History representation and the approximation comparator

  • For sufficiently large p, a deterministic comparator approximates the target on both discrete and continuous histories, depends on the target and budget rather than the sample, and can be clipped without increasing its error.Clipping is a fixed, parameter-free output operation used for analysis and final prediction, not an added learned layer or part of the main theorem’s training objective.Source: §G.1 · p. 67

G.2 Uniform stability and covering numbers

  • The predictor class has a length-independent parameter cover because normalized attention averages have parameter-bounded sensitivity, yielding a cover without dependence on context length or the modulus.The cover is uniform over history measures and applies before and after clipping.Source: §G.2 · pp. 67–68

G.4 Raw empirical minimization followed by clipping

  • Raw empirical minimization followed by clipping is justified using concentration for the clipped class and a single bounded raw comparator, rather than uniform concentration over raw losses.A measurable empirical minimum exists, but exhaustive fine-grid search is only an existence argument; the proof gives no global-optimality guarantee for SGD.Source: §G.4 · pp. 70–71
  • The conditional test-risk bound retains the logarithmic root rate when τopt = O(qβ(N)^2) at fixed confidence, independently of the modulus, length law, and maximum length.The proof combines concentration for the clipped class with one bounded raw comparator; it does not require uniform concentration for all raw predictors.Source: §G.4 · pp. 70–71

G.5 Sampling variants and scope

  • Continuity alone cannot ensure a uniform learning rate: for some bounded continuous teachers and sampling designs, every learner has expected prediction risk at least 1/2.The construction uses length-one sequences with 2N distinct tokens and independent random labels; a quantitative restriction such as the stated β-smooth target condition is sufficient, not necessary for learnability.Source: §G.5 · p. 71
  • The proof extends to iid histories and fully observed labeled paths under conditional-mean and bounded-label assumptions, but risk is controlled only under the induced history law, not uniformly or under arbitrary distribution shift.Each complete sequence has total weight one, lengths need no moment bound, and overlapping windows are not independent observations; sparse observation or approximate quadrature adds error terms. The result concerns bounded squared regression with conditional centering and the specified β-smooth seminorm, not unrestricted cross-entropy or generic Fréchet smoothness, and gives no statistical lower bound.Source: §G.5 · pp. 71–72

H Empirical protocol and full results

  • The appendix records sampling, pretrained maps, scale sensitivity, and refinement checks, while its measurements characterize finite-scale, representation-dependent regularity rather than a common asymptotic Hölder class.It includes raw physical-signal baselines alongside continuous-content and layer-wise text experiments, and contrasts continuous patch embeddings with BigBird text input embeddings.Source: H Empirical protocol and full results · p. 72

H.1 Estimator interpretation and sensitivity

  • Finite-band fitted slopes are encoding- and RMS-dependent estimates, not a common Hölder constant or certification of regularity.The exact grid Hölder seminorm takes the maximum over every lag; low R2 weighs against a single power law, and correlated lag points do not support regression-standard-error uncertainty across recordings.Source: §H.1 · pp. 72–73

H.2 Data and sampling

  • The benchmark samples six scalar series, ECG, ground motion, and four text controls using fixed domain-specific windows and resampling rules.Scalar replicates are windows rather than independent sources; seismic acquisition metadata are incomplete, and the continuous-content comparison uses only the six scalar series.Source: §H.2 · p. 73

H.3 Pretrained representations

  • BigBird input embeddings use learned word, token-type, and interpolated position vectors, whereas Chronos-Bolt uses a continuous residual ReLU patch map.The BigBird comparison is input-only, without attention or fine-tuning; Bolt’s window-dependent normalization does not establish a common theorem-level Hölder prefactor.Source: §H.3 · pp. 73–74

H.4 The continuous-content comparison

  • Dense and native Chronos-Bolt patch outputs match exactly at corresponding endpoints, while shuffled controls have near-zero mean slopes and unprojected delay patches retain positive slopes.Input-shuffled means are 0.005–0.035 and output-shuffled means are within 0.003 of zero; these are one-per-window controls, not significance tests.Source: §H.4 · pp. 74–75
  • A fixed Lipschitz patch map preserves an upper Hölder bound for fixed physical offsets, but does not imply equal exponents or a shared constant across resolutions.Fixed sample-count patches change physical support across resolutions; native and dense primary fits also differ in finest scales, and Beijing and traffic lack enough native fine-band lags for estimates.Source: §H.4 · pp. 74–75

H.5 Raw physical-signal baselines

  • 0.041–0.389 is the range of mean maximum-increment slopes across the raw physical-signal baselines, with variability and fit quality qualifying their interpretation.The regressions use h ≤ 1/32; the reported slopes are not interchangeable with weaker RMS or percentile increment statistics.Source: §H.5 · pp. 75–76, Table 4 · p. 75
  • Jena’s percentile sensitivity slope is 0.253 versus 0.421 for its maximum-increment slope, illustrating that percentile statistics cannot replace the theory’s uniform maximum.A common normalized cutoff spans different physical durations and different numbers of fine lags across signals.Source: §H.5 · pp. 75–76

H.6 Calibration and direct refinement checks

Synthetic calibration shows roughness ordering but patch encoding can alter jump scaling; nested-grid checks show limited prefactor stability, while added text sources do not reveal a more regular BigBird input regime.

H.6 Calibration and direct refinement checks

  • The calibration uses 12 realizations per synthetic scenario, including fBm with H = 0.25, 0.50, and 0.75, plus smooth, noise, and jump controls.For fBm, H is the supremal pathwise exponent: paths are Hölder for every α < H, not generally at H itself.Source: §H.6 · p. 76
  • Dense Bolt slopes track the fBm roughness ordering, but encode the jump with a positive slope (0.166 ± 0.017) versus −0.012 ± 0.005 before encoding.On matched raw lags of at least 16, the dense jump slope is −0.013 ± 0.006, exposing a scale crossover rather than an asymptotic certificate.Source: §H.6 · p. 76, Figure 2 · p. 76
  • At α = 0.25, mean A513/A65 is 1.045 for dense traffic and 1.129 for Beijing, while dense weather and appliances grow by factors 1.626 and 1.674.These nested-grid checks show prefactor stability for some representations, but do not establish a shared 0.25-Hölder class; the finite cadence and central subinterval limit their scope.Source: Figure 3 · p. 77, §H.6 · p. 77

H.7 A structured-text corpus screen

  • The text screen tests Shakespeare, the King James Bible, and Python using the same frozen BigBird input map, with paired content-ID shuffling controls.Sources and windows were chosen before measuring increments, and neither the new texts nor Python stream is periodically repeated.Source: §H.7 · p. 77
  • Across the three preselected text additions, maximum-slope means are 0.0106, 0.0115, and 0.0124, none above the original IMDb mean of 0.0130.Shuffling gives similarly small values, and RMS slopes remain between 0.0148 and 0.0191 across all seven sources; this is a finite-scale result for one input pipeline, not every text representation.Source: Table 5 · p. 78, §H.7 · p. 78

H.8 Text representations across transformer depth

BigBird shows a modest, non-monotone intermediate-layer gain in effective maximum-increment slopes, while the gain does not persist to the final block. Fixed-context experiments characterize depth-dependent geometry but do not establish refinement-asymptotic Hölder improvement.

Matched windows and native inference

  • The diagnostic uses frozen BigBird and DistilGPT2 at native contexts, with the same windows and lags across depths and no training, smoothing, or positional interpolation.Depth zero is the embedding representation; DistilGPT2’s final LayerNorm is recorded separately from block six.Source: §H.8 · pp. 78–79

An intermediate-layer gain in BigBird

  • Block eight improves on the input in all eight retained windows of each source on the primary band, with positive mean gain also on broader bands.Not every short-band paired difference is positive; peak depth is descriptive rather than selected to tune the fitting band.Source: §H.8 · p. 79
  • BigBird’s mean primary slope rises from 0.013–0.014 at input to 0.028–0.033 around blocks seven and eight, then returns to 0.002–0.014 at the final block.Table 6 reports mean ± sample SD across eight matched natural-order windows per source on h ≤1/32; SD is descriptive, not a confidence interval.Source: Table 6 · p. 79, §H.8 · p. 79

Causal representations and average-increment scaling

  • Shuffled BigBird inputs retain block-eight slopes of 0.0363, 0.0382, 0.0354, and 0.0291, so the intermediate gain cannot be attributed solely to meaningful word order.Final-layer centered token RMS is 0.55–0.70 times input, while the adjacent maximum is 0.82–0.90 times input; smaller increments need not indicate a larger scaling exponent.Source: §H.8 · pp. 79, 81

Comparing multiscale shape across depth

  • Figure 4 overlays normalized mean worst-increment curves on shared log–log axes; intermediate blocks are steeper, but the gain does not persist to the final block.The displayed differences can be small: curves often differ by only 10–20% across hundreds of token lags.Source: §H.8 · p. 79, Figure 4 · p. 80

Causal representations and average-increment scaling

  • For DistilGPT2, RMS slopes show a stronger depth effect than maximum slopes, and natural order improves this average statistic except on WikiText.RMS input means are 0.0016–0.0062, final means are 0.0216, 0.0393, 0.0569, 0.0661, and shuffled means are 0.0231, 0.0218, 0.0293, 0.0347; RMS gain does not establish the uniform increment control used in the theory.Source: §H.8 · p. 81, Figure 5 · p. 82

Sensitivity and scope

  • These fixed-context experiments support depth-dependent effective regularity, including an intermediate gain, but not asymptotic Hölder improvement under refinement, a common all-resolution bound, or prediction-quality claims.After removing the first and last 64 content positions, final maximum-slope means remain between 0.002 and 0.020 for BigBird and −0.001 and 0.017 for DistilGPT2.Source: §H.8 · p. 81
Loading 2609.35055v1…