PaperlayerView exact revision on arXiv

Source-linked AI summary

Stability of Measure-to-Measure Transformers on Sub-Gaussian Data

Frank Cole, Nicholas H. Nelsen, Takashi Furuya

arXiv:2610.07717v1stat.MLcs.LG

TL;DR

The paper addresses the limited mathematical understanding of transformer measure-to-measure operators, especially on inputs with unbounded support. It proves stability and regularity results for sub-Gaussian data, estimates empirical approximation errors, and extends the analysis to cross-attention and measure-to-measure approximation. The results also identify limits involving layer normalization and the depth of the cross-attention analysis.

  • Problem

    The mathematical properties of transformer measure-to-measure operators on unbounded-support inputs require analysis, since softmax needs tail decay to be well-defined.

    Source: §1 · p. 1, §3.1 · p. 5

  • Method

    The paper analyzes mean-field transformer operators on sub-Gaussian measures, proving stability and Hölder regularity and applying these results to empirical measures, cross-attention, and approximation.

    Source: §2.1 · p. 4, §3.2 · p. 5, §3.3 · p. 6

  • Results

    Transformers preserve sub-Gaussianity and satisfy Hölder and empirical-error bounds, while cross-attention has different regularity in its two inputs and the paper establishes sample-complexity and approximation guarantees.

    Source: §3.4 · pp. 7–8, §4 · p. 8, §3.5 · p. 8

  • Takeaways & Limitations

    The results provide stability and finite-sample theory for measure-to-measure transformers on sub-Gaussian data.

    Source: §4 · p. 8

    Layer normalization breaks continuity, so developing compatible analogs of the results remains future work.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Transformers have exhibited impressive empirical success across various domains, but their theoretical foundations remain less developed. This work constitutes a mathematical study of the measure-to-measure operators defined by transformers. We show that transformers map sub-Gaussian inputs to sub-Gaussian outputs; this ensures that taking arbitrary-length compositions of the softmax operator is well-defined. We then show that transformers are H\"older continuous with respect to the 1-Wasserstein distance on appropriate spaces of sub-Gaussian inputs. This allows us to establish estimates on the error propagation along a transformer between a sub-Gaussian input and its empirical approximation. We also study a mean-field analog of the cross-attention mechanism, which is an operator from a pair of probability measures to a single probability measure. We show that cross-attention exhibits different H\"older regularity and sample-complexity in its two input arguments. Last, we apply our results to deduce approximation guarantees for measure-to-measure transformers. Together, these results provide a firm stability and finite-sample theory for transformers on sub-Gaussian data.

1 INTRODUCTION

The paper develops a measure-theoretic analysis of transformers, focusing on measure-to-measure operators and inputs with unbounded support. Its contributions establish stability, regularity, sample-complexity, cross-attention, and approximation results for these operators.

1 INTRODUCTION

  • The analysis treats transformer outputs as functions of empirical input-token distributions and studies measure-to-measure operators, particularly for unbounded-support inputs.Sub-Gaussianity is presented as a sufficient tail condition for stability that still accommodates distributions with unbounded support.Source: §1 · pp. 1–2

1.1 Main Contributions

  • Arbitrarily deep mean-field transformers map sub-Gaussian measures to sub-Gaussian measures, keeping the softmax normalizing constant finite across layers.The paper identifies sub-Gaussianity as a sufficient condition for well-defined outputs on measures with unbounded support.Source: §1.1 · p. 2
  • On appropriate subsets of sub-Gaussian measures, the transformer pushforward operator is Hölder continuous in the 1-Wasserstein metric.The paper uses this regularity to estimate the error between a sub-Gaussian measure and its empirical counterpart after transformation.Source: §1.1 · p. 2
  • The paper estimates the 1-Wasserstein error between transformer outputs on a sub-Gaussian measure and its empirical counterpart, with the bound depending on the operator’s Hölder exponent.It also proves concentration inequalities for the sub-Gaussian parameters of empirical measures.Source: §1.1 · p. 2
  • Mean-field cross-attention is jointly Hölder continuous with different regularity in its two input variables, with a sample-complexity result for compactly supported µ and sub-Gaussian η.The mapping takes a pair of measures to a measure, and the paper specifies the assumptions for its sample-complexity result.Source: §1.1 · p. 2
  • Combining sample-complexity estimates with recent approximation results yields universal approximation of continuous measure-to-measure operators on certain W1-compact sets, even when transformers receive only input samples.The stated result applies to certain W1-compact subsets of probability measures.Source: §1.1 · p. 2

1.2 Related Work

2 BACKGROUND

The background formulates attention and transformer layers as maps on probability measures, then defines the sub-Gaussian input class used in the analysis. It also notes that layer normalization is outside the paper’s continuity analysis.

2.1 Mean-Field Transformers

  • Mean-field self-attention extends sequence attention to probability measures, recovering traditional attention for empirical measures when the defining integrals converge.A token’s output depends on the input sequence through its empirical measure, and an attention head computes a weighted average of transformed tokens.Source: §2.1 · pp. 3–4
  • A mean-field transformer composes multi-head self-attention and feedforward layers using an associative operation on in-context maps.The feedforward layers are ReLU networks; restricting them to be shallow loses no generality because attention layers can represent the identity map.Source: §2.1 · p. 4
  • The analysis omits normalization layers because layer normalization breaks transformer continuity; developing compatible results is left for future work.The authors note relevance to post-LN transformers that accommodate unbounded support at the initial input layer.Source: §2.1 · p. 4
  • A mean-field transformer induces a measure-to-measure operator by pushing its input measure through the transformer, and composition of these operators matches transformer composition.This correspondence holds when the composed in-context mapping is well-defined.Source: §2.1 · p. 4

2.2 Sub-Gaussian Probability Distributions

  • Sub-Gaussian measures are defined through square-exponential tail decay and generalize Gaussian tail behavior.The paper identifies this class with the classical sub-Gaussian definition based on finite sub-Gaussian norm.Source: §2.2 · pp. 4–5

3 MAIN RESULTS

The results establish stability and finite-sample guarantees for measure-to-measure transformers on sub-Gaussian data, then use them to prove approximation guarantees. The analysis also treats cross-attention, whose regularity and sample complexity differ across its two inputs.

3 MAIN RESULTS

  • The paper states its bounds in the 1-Wasserstein metric and leaves extension to Wp for p > 1 as future work.The authors expect such extensions to hold up to constants depending on p.Source: §3 · p. 5

3.1 Transformers Preserve Sub-Gaussianity

  • Mean-field transformers send sub-Gaussian measures to sub-Gaussian measures, so their outputs remain in the required class even under arbitrary-depth compositions.Sub-Gaussian self-attention grows at most affine-linearly, and the pushforward by a transformer preserves sub-Gaussianity.Source: §3.1 · p. 5

3.2 H¨older Regularity of Transformers on Sub-Gaussian Domains

  • On fixed sub-Gaussian domains, measure-to-measure transformer operators are Hölder continuous in the 1-Wasserstein distance, including general transformers.The self-attention estimate gives a possibly sub-optimal exponent; extending it to general transformers sacrifices control of that exponent.Source: §3.2 · p. 6

3.3 Sample Complexity of Transformers

  • The expected 1-Wasserstein error between a transformer applied to a sub-Gaussian measure and to its iid empirical measure is bounded using the transformer’s Hölder exponent.The proof first shows that the empirical measure belongs to a suitable sub-Gaussian class with high probability, then combines Hölder regularity with a Wasserstein law of large numbers.Source: §3.3 · p. 6

3.4 Transformers With Mean-Field Cross-Attention

  • Cross-attention has different regularity and sample-complexity behavior in its two input measures.Its measure-to-measure operator is smoother in the second argument, while the stated sample-complexity results cover compactly supported first inputs with sub-Gaussian second inputs, or compact support for both.Source: §3.4 · pp. 7–8

3.5 Consequences for Measure-to-Measure Approximation With Transformers

  • With sufficiently many samples, transformer operators universally approximate continuous measure-to-measure operators on qualifying uniformly sub-Gaussian compact sets.The set must be absolutely continuous with respect to a common non-atomic reference measure; for a fixed approximating transformer, sampling error vanishes at a polynomial rate up to logarithmic factors.Source: §3.5 · p. 8

4 CONCLUSION

The paper establishes stability and finite-sample results for measure-to-measure transformers and identifies open directions. It also reports limited use of generative AI tools in developing some proof ingredients.

4 CONCLUSION

  • Transformers map sub-Gaussian measures to sub-Gaussian measures, are Hölder continuous in the 1-Wasserstein distance on appropriate domains, and admit empirical error-propagation estimates.The paper also provides analogous results for cross-attention.Source: §4 · p. 8
  • The sample-complexity results have a curse of dimensionality; whether transformer parametric complexity or low-dimensional structure can improve the rates remains open.The paper also leaves non-iid sampling and generalization-error analysis for supervised learning on probability measures to future work.Source: §4 · p. 8

AI Use Statement

  • The authors used generative AI to formulate mathematical claims and provide proof ingredients for Proposition 3.2 and Lemma A.1, and reviewed all AI-assisted work.They report not using it for writing proofs, proposing hypotheses, designing methodology, or interpreting results.Source: AI Use Statement · p. 9

CHECKLIST

The checklist marks theoretical assumptions and proofs as included, while empirical-results, asset, and human-subjects items are marked not applicable.

CHECKLIST

  • The checklist marks clear explanations of assumptions and complete proofs of theoretical results as included.It also marks the mathematical setting, assumptions, and model description as included.Source: CHECKLIST · p. 11
  • Items for reproducing empirical results, training details, result statistics, and computing infrastructure are marked not applicable.Source: CHECKLIST · pp. 11–12
  • Checklist items on existing or new assets and on crowdsourcing or human subjects are marked not applicable.Source: CHECKLIST · p. 12

A PROOFS

The appendix collects proofs of results presented in the main text.

A PROOFS

  • The appendix collects proofs of results presented in the main text.Source: A PROOFS · p. 13

A.1 Proofs for Section 3.1: Transformers Preserve Sub-Gaussianity

The proof bounds the growth of a mean-field self-attention map on sub-Gaussian inputs by analyzing a convex cumulant generating function. It handles small and large arguments separately to establish affine-linear growth.

A.1 Proofs for Section 3.1: Transformers Preserve Sub-Gaussianity

  • The proof reduces to a single attention head without a skip connection and expresses the attention map using the cumulant generating function.It then uses convexity to target affine-linear growth of the cumulant generating function's gradient.Source: §A.1 · p. 14
  • For arguments with norm at most 1, the cumulant generating function's gradient is uniformly bounded using the sub-Gaussian parameters.The proof bounds the numerator and denominator separately.Source: §A.1 · p. 14
  • For arguments with norm at least 1, sub-Gaussian growth bounds and convexity yield affine-linear growth of the attention map, with constants depending on the input's sub-Gaussian parameters and attention matrices.Source: §A.1 · p. 15

A.2 Proofs for Section 3.2: H¨older Regularity of Transformers on Sub-Gaussian Domains

The proof bounds transformer stability by truncating sub-Gaussian inputs and controlling the resulting errors. It then extends Hölder continuity from a single layer to compositions by induction.

A.2 Proofs for Section 3.2: H¨older Regularity of Transformers on Sub-Gaussian Domains

  • Truncation reduces the stability proof to a bounded-input estimate plus a tail error; the multi-head bound follows from the single-head estimate.The truncation estimate controls the difference between Γ(µ, x) and Γ((ΠR)#µ, x).Source: §A.2 · p. 15
  • The single-head proof decomposes the attention-output difference into numerator and denominator terms, then bounds the resulting sub-Gaussian integrals.The proof uses a uniform lower bound on the denominator and a mean-value bound for the numerator.Source: §A.2 · pp. 15–16
  • The final stability estimate is Hölder, and induction extends this regularity across compositions of attention and ReLU feedforward layers.At each induction step, the preceding composition maps the input into another sub-Gaussian class, where the next layer is Hölder continuous.Source: §A.2 · p. 18

A.3 Proofs for Section 3.3: Sample Complexity of Transformers

The proofs control empirical approximation by separating sampling error from the effect of replacing the input measure. They also establish a high-probability sub-Gaussian bound for the empirical input.

A.3 Proofs for Section 3.3: Sample Complexity of Transformers

  • P(AN) ≥ 1 − 2αN^-2 for N ≥ Nα,β, where AN is the event that µN lies in SG3+α/7,7β.The proof uses this event to apply the transformer’s Hölder estimate to the empirical measure.Source: §A.3 · p. 19
  • The general sample-complexity proof splits the output error into sampling and measure-replacement terms, bounding each separately.The proof estimates the first term using a coupling and the second using the transformer’s regularity.Source: §A.3 · pp. 19–20
  • E[W1(T(µ, · )#µ, T(µN, · )#µN)] ≤ CR,T E[W1(µ, µN)] for inputs supported in BR.On P(BR), the measure-to-measure map is Lipschitz, with constant depending on R and the parameters of T.Source: §A.3 · p. 20

A.4 Proofs for Section 3.4: Transformers With Mean-Field Cross-Attention

The cross-attention proof controls changes in its two input measures separately. Its stability estimate has a Hölder-type term for one input and a linear Wasserstein term for the other.

A.4 Proofs for Section 3.4: Transformers With Mean-Field Cross-Attention

  • W1(Γ(µ, · )#µ, Γ(µ0, · )#µ0) ≤ C log(W1(µ, µ0)^-1)^3/2(W1(µ, µ0)^γ + W1(η, η0)).Here γ is defined from k1 and k2 in Equation (A.3).Source: §A.4 · p. 22
  • For one bounded input and one sub-Gaussian input, the proof bounds the output error by separating changes in the two arguments.It uses truncation and Lipschitz bounds to control the separate terms.Source: §A.4 · p. 23
  • When both inputs are bounded, the proof bounds terms using covering-number estimates for function classes associated with multi-head attention.The proof applies Dudley’s chaining estimate and covering-number bounds for the relevant classes.Source: §A.4 · pp. 23–24

A.5 Proof of Theorem 3.11

The approximation proof combines an approximating transformer with the paper’s stability result. The appendix also collects auxiliary lemmas supporting the main proofs.

A.5 Proof of Theorem 3.11

  • The proof uses an approximating transformer Tϵ from Furuya et al. (2026b), then combines its approximation bound with Theorem 3.7 through the triangle inequality.Source: §A.5 · p. 25

B AUXILIARY LEMMAS

B.1 Properties of Sub-Gaussian Measures

These appendices collect stability and tail estimates for sub-Gaussian measures, including compactness, controlled pushforwards, and empirical-measure bounds. They also use truncation to obtain Wasserstein convergence estimates.

B.1 Properties of Sub-Gaussian Measures

  • Sub-Gaussian measure classes are W1-compact, with first-moment control also recorded.The section characterizes W1 convergence through weak convergence together with convergence of first moments.Source: §B.1 · pp. 25–26
  • A map with at-most-linear growth sends a sub-Gaussian measure to a sub-Gaussian measure with adjusted parameters.For ∥T(x)∥≤a+b∥x∥, the stated parameters are eα=αe^(a²/(b²β)) and eβ=2b²β.Source: §B.1 · p. 26
  • With probability at least 1−2αN^(1−ρ), an empirical measure from sub-Gaussian data satisfies the proposition’s stated sub-Gaussian bound for sufficiently large N.The proposition assumes ρ>1 and k>2ρ.Source: §B.1 · p. 29

B.2 Gaussian and Sub-Gaussian Tail Bounds

  • Tail estimates establish that truncating sub-Gaussian measures is uniformly accurate and control tails of first moments and modulated exponential moments.These estimates are used in the appendix’s subsequent continuity arguments.Source: §B.2 · p. 32

B.3 Properties of Transformers

The appendix develops stability estimates for attention and transformer operators, then supplies Wasserstein convergence and covering-number bounds used in the analysis. It also gives a support-dependent bound for transformers applied to finite measures.

B.3 Properties of Transformers

  • Self-attention estimates yield Wasserstein stability bounds for the corresponding measure-to-measure operators.The appendix also records explicit Lipschitz estimates for multi-head self-attention on bounded domains.Source: §B.3 · p. 35
  • For a transformer applied to an empirical measure, its in-context map is bounded in terms of the support radius, with constants that may grow exponentially in the number of layers.The stated bound applies to measures supported on finitely many points.Source: §B.3 · pp. 35–36

B.4 Wasserstein Law of Large Numbers

  • For d≥3, Proposition B.18 asserts a constant C_{d,α,β} depending only on d, α, and β, with a cited rate fragment N^(-1/d) for all N∈N.Source: §B.4 · p. 38

B.5 Covering Number Estimates

  • The appendix presents covering-number estimates for function classes associated with multi-head attention to control entropy integrals.It includes a main upper bound for those associated classes.Source: §B.5 · p. 39

Related papers

Loading 2610.07717v1…