PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Approximation of Measure-to-Measure Operators by Pushforwards

Takashi Furuya, Nicholas H. Nelsen, Frank Cole

arXiv:2609.35483v1cs.LGcs.NAmath.NAstat.ML

TL;DR

The paper studies when continuous operators between probability measures can be approximated by continuous, measure-dependent pushforwards, despite deterministic pushforwards’ inability to redistribute atomic mass. It introduces the uniform level set condition and proves uniform approximation on compact families satisfying it, with consequences for measure-theoretic transformers and continuously varying source measures.

  • Problem

    The paper asks when arbitrary continuous operators between probability measures can be approximated by continuous measure-dependent pushforwards, since deterministic pushforwards cannot split or redistribute atomic mass.

    Source: §1 · pp. 1, 3

  • Method

    The uniform level set condition uses a continuous measure-dependent scalarization whose shrinking level-set neighborhoods carry uniformly vanishing mass across the compact input family.

    Source: §3.1 · p. 5, §3.2 · p. 6

  • Results

    Every continuous operator with finite p-th moment outputs is uniformly approximable in p-Wasserstein distance on compact input families satisfying the uniform level set condition.

    Source: §3.2 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    The approximation theorem yields universal approximation by measure-theoretic transformers on qualifying compact families and extends to continuously varying source measures relevant to cross-attention.

    Source: §5 · p. 9, §4.2 · p. 9

    The sufficient uniform level set condition may not be necessary, and approximation of noncontinuous operators such as the joint-to-conditional map is beyond the paper’s scope.

    Source: §5 · p. 9

Abstract

from arXiv · show

Many learning tasks map an input distribution to an output distribution. A natural way to model such an operator is to transform each input sample using a continuous function that may depend on the entire input distribution, and then take the distribution of the transformed samples. This defines a measure-dependent pushforward model and includes measure-theoretic formulations of transformers. We ask when such models can approximate arbitrary continuous operators between spaces of probability measures. We first show that universal approximation fails when atomic inputs are allowed: some continuous measure-to-measure operators that split or redistribute atomic mass cannot be approximated arbitrarily well by deterministic pushforward models. We then introduce the uniform level set condition, which requires a continuous measure-dependent scalarization whose shrinking level set neighborhoods carry uniformly vanishing mass over the input family. This condition is satisfied, in particular, by compact families of absolutely continuous measures. On every compact family satisfying this condition, we prove that any continuous measure-to-measure operator with outputs of finite $p$-th moment can be uniformly approximated, in the $p$-Wasserstein distance, by continuous measure-dependent pushforwards. Combining our theorem with existing approximation results for measure-dependent in-context maps yields universal approximation by measure-theoretic transformers. We also extend the framework to continuously-varying source measures, yielding a corresponding universality result for a class of pushforward models that are closely aligned with cross-attention architectures.

1 INTRODUCTION

The paper asks when continuous measure-to-measure operators can be approximated by continuous, measure-dependent pushforwards. It identifies an obstruction from atomic inputs and establishes approximation results under the uniform level set condition, including for measure-theoretic transformers.

1 INTRODUCTION

  • The central question is whether arbitrary continuous operators between probability measures can be approximated by continuous pushforwards whose transformations depend on the input measure.The motivating applications include Bayesian prior-to-posterior maps, Wasserstein proximal maps, and McKean–Vlasov solution maps.Source: §1 · p. 1
  • The uniform level set condition requires a continuous measure-dependent scalarization whose shrinking level-set neighborhoods have uniformly vanishing mass across the input family.It is satisfied by compact families of non-atomic measures in one dimension, families dominated by a common non-atomic reference measure, and certain continuously varying singular families.Source: §1 · p. 2
  • On compact input families satisfying that condition, every continuous operator with finite p-th moment outputs is uniformly approximable in p-Wasserstein distance by continuous measure-dependent pushforwards.The condition is sufficient and necessarily excludes atomic measures.Source: §1 · p. 2
  • Combining the approximation theorem with existing in-context-map universality results yields measure-theoretic transformer universality on qualifying compact families, with a further extension to continuously varying source measures relevant to cross-attention.Measure-theoretic transformers induce pushforward maps by transforming tokens through a measure-dependent function.Source: §1 · pp. 1–3
  • Deterministic pushforwards are not universal on general compact input families because they cannot split or freely redistribute atomic mass.The paper gives continuous operators that remain a strictly positive 1-Wasserstein distance from every deterministic pushforward model.Source: §1 · pp. 1–3

2 THE PUSHFORWARD APPROXIMATION PROBLEM

This section formulates approximation of continuous measure-to-measure operators by continuous measure-dependent pushforwards and shows why atomic inputs obstruct universality. The obstruction covers both splitting an atom and changing weights while retaining the same support cardinality.

2.1 THE MEASURE-TO-MEASURE APPROXIMATION PROBLEM

  • The section asks whether continuous measure-to-measure operators can be uniformly approximated by continuous pushforwards whose transformation depends on the input measure.The approximation setup considers compact input families and continuous target operators; pushforward models are also motivated by computational considerations.Source: §2.1 · p. 3
  • Bayesian updating gives a continuous prior-to-posterior map that generally changes atomic weights, so it need not admit an exact deterministic pushforward representation.The example assumes a continuous negative log-likelihood bounded from below for continuity of the map.Source: §2.1 · p. 4

2.2 OBSTRUCTIONS CAUSED BY ATOMIC MEASURES

  • When atomic inputs are allowed, there are compact families and continuous target operators that no continuous deterministic pushforward approximates to arbitrary accuracy.The section states this obstruction as Proposition 2.2.Source: §2.2 · p. 4
  • A deterministic pushforward cannot approximate splitting one input atom into two, and the obstruction persists when input and output have the same support cardinality but different weights.Thus the limitation is not merely a change in the number of atoms; deterministic pushforwards also cannot generally modify their weights.Source: §2.2 · pp. 4–5

3 UNIVERSAL APPROXIMATION BY CONTINUOUS PUSHFORWARD MODELS

The uniform level set condition identifies compact input families on which continuous measure-dependent pushforwards can approximate arbitrary continuous measure-to-measure operators. It includes non-atomic and some singular families, while the main construction uses an exact continuous map to a common uniform distribution.

3.1 UNIFORM LEVEL SET CONDITION

  • The uniform level set condition requires a continuous scalarization whose level sets carry uniformly vanishing mass, and it enables an exact continuous uniformizer.The condition is equivalent to the scalar pushforwards forming a non-atomic family.Source: §3.1 · p. 5
  • Every compact family of non-atomic probability measures on the real line satisfies the condition, which characterizes such families there.The condition also implies that each measure in the family is non-atomic.Source: §3.1 · p. 5
  • Compact families dominated by a common non-atomic reference measure satisfy the condition, including families with densities on a fixed low-dimensional manifold.Lebesgue absolute continuity is not required when a suitable atomless reference measure is available.Source: §3.1 · p. 6
  • The condition also covers normalized arclength measures on certain compact families of injective curves, despite their singular supports varying continuously.The stated example additionally assumes the curves are uniquely determined by their images.Source: §3.1 · p. 6

3.2 MAIN RESULT

  • Theorem 3.6 gives uniform p-Wasserstein approximation of any continuous operator on a compact family satisfying the uniform level set condition.The proof uses a continuous in-context map that pushes every input measure exactly to Unif[0, 1].Source: §3.2 · pp. 6–7
  • The proof approximates outputs through truncation, finite nets, continuous weights, and smoothed transitions, with the triangle inequality combining the errors.The paper also interprets the theorem as an operator analogue of diagonal approximation when the source measure varies over a compact family.Source: §3.2 · pp. 6–7

4 APPLICATIONS TO TRANSFORMER ARCHITECTURES

The paper applies pushforward approximation to measure-theoretic transformers and extends the framework to continuously varying source measures. This yields universality under the uniform level set condition for self-attention, and a corresponding result for a specified cross-attention configuration.

4.1 MEASURE-THEORETIC TRANSFORMERS

  • Measure-theoretic attention acts on a token while depending on the input measure, and empirical-measure inputs recover the usual finite-sum attention mechanism.The formulation also applies to arbitrary probability measures, including non-atomic measures, without fixing the token count in advance.Source: §4.1 · p. 8
  • Measure-theoretic transformers universally approximate continuous measure-to-measure operators on compact input families satisfying the uniform level set condition.This result combines the pushforward approximation theorem with universality results for measure-dependent in-context maps.Source: §4.1 · p. 8
  • The transformer universality result assumes the uniform level set condition, while the proof truncates inputs before evaluating self-attention layers.The theorem permits full-support input measures even though its construction evaluates attention at compactly supported measures.Source: §4.1 · p. 8

4.2 CONTINUOUSLY-VARYING SOURCE MEASURES AND CONNECTION TO CROSS-ATTENTION

  • For continuously varying source measures, the approximation theorem allows the transport source to differ from the input measure, matching a cross-attention setting.The query-token measure may differ from the context measure and may live on another ambient space.Source: §4.2 · p. 9
  • The cross-attention universality argument covers source tokens attending to context tokens with token-wise source processing, but not architectures that also use source self-attention.Covered source measures include a fixed non-atomic measure and an absolutely-continuously smoothed input measure.Source: §4.2 · p. 9

5 CONCLUSION

The conclusion identifies atomic measures as an obstruction to universal pushforward approximation and presents the uniform level set condition as a sufficient regime, weaker than absolute continuity. It also notes open questions about necessity, discontinuous operators, and quantitative transformer bounds.

5 CONCLUSION

  • Deterministic pushforwards cannot split or adjust atomic weights, while the uniform level set condition enables uniform approximation on compact sets and permits measures with low-dimensional support.The condition is weaker than absolute continuity and implies a uniform approximation theorem for measure-theoretic transformers.Source: §5 · p. 9
  • The condition's necessity is unknown; approximation of discontinuous operators such as the joint-to-conditional map and quantitative transformer bounds remain open.The cited joint-to-conditional operator may be discontinuous, placing its approximation beyond this paper's scope.Source: §5 · p. 9
  • The authors report using generative AI for proof assistance, claim formulation, editing, and literature searches, with mathematical claims and proofs independently checked.They state that AI was not used to develop theoretical models or design methodology.Source: AI USE STATEMENT · p. 10

Supplementary Material for: Universal Approximation of Measure-to-Measure Operators by Pushforwards

The supplementary material contains auxiliary results and supporting material.

Supplementary Material for: Universal Approximation of Measure-to-Measure Operators by Pushforwards

A.1 ADDITIONAL MATERIAL FOR SECTION 2

The appendix gives additional support for the atomic-input obstruction using a continuous Wasserstein proximal map, and clarifies an equivalent characterization of the uniform level set condition. The latter holds precisely when the scalarized measures are non-atomic throughout the compact input family.

A.1 ADDITIONAL MATERIAL FOR SECTION 2

  • The W1 proximal map Fτ is continuous, yet maps an atomic Dirac input to a non-atomic measure that no deterministic pushforward of that input can represent.The appendix constructs the map through an entropy-based variational problem with a unique minimizer.Source: §A.1 · pp. 13–14

A.2 ADDITIONAL MATERIAL FOR SECTION 3

  • For a compact input family and continuous scalarization φ, the uniform level set condition is equivalent to φ(µ, ·)#µ being non-atomic for every µ in the family.Source: §A.2 · p. 15
  • The appendix proves the equivalence by showing that the scalarized measures vary continuously and form a compact family in the weak topology.Source: §A.2 · pp. 15–16

A.3 ADDITIONAL MATERIAL FOR SECTION 4

The appendix extends the approximation result to compact input families by Gaussian smoothing, including families with discrete measures. It also records a projection result used in proving an example of the uniform level set condition.

A.3 ADDITIONAL MATERIAL FOR SECTION 4

  • Gaussian smoothing lets the approximation theorem cover any Wp-compact input family, including singular or atomic measures, without requiring the uniform level set condition on the original family.For σ > 0, the smoothed family lies in the absolutely continuous measures and can be treated using Theorem 4.4.Source: §A.3 · pp. 16–17
  • The appendix states universality of deep transformer in-context maps for compact measure families on Rd and inputs restricted to a compact token domain Ω.Source: §A.3 · p. 17

B.2 PROOF OF EXAMPLE 3.4

  • For a non-atomic measure on Rd, almost every one-dimensional projection is non-atomic.Source: §B.2 · p. 20

B.3 PROOF OF EXAMPLE 3.5

The proof constructs a continuous scalarization for a compact family of measures induced by curves. Under the condition that distinct curves have distinct images, this family satisfies the uniform level set condition.

B.3 PROOF OF EXAMPLE 3.5

  • The map from a curve and parameter to the curve’s value is a homeomorphism onto its image, making that image compact.The proof uses compactness of the curve family and injectivity of each curve.Source: §B.3 · p. 21
  • Compactness of K′ in C1([0, 1], Q) gives a finite uniform speed bound C for every γ ∈ K′ and s ∈ [0, 1], while the assumption gives curve lengths at least c for every γ ∈ K′.Source: §B.3 · p. 21
  • Distinct curves having distinct images makes the map from each curve to its induced measure injective, with a continuous inverse on the measure family.This allows the curve-based construction to be defined in terms of the measures.Source: §B.3 · p. 22
  • The proof transfers the curve’s parameter to a continuous scalarization on the measure family and concludes that the family satisfies the uniform level set condition.The scalarization is defined using the inverse curve-to-measure map and the extended parameter map.Source: §B.3 · p. 22

B.4 PROOF OF THEOREM 3.6

The proof of Theorem 3.6 builds a continuous pushforward approximation by truncating the output, discretizing it, and regularizing the resulting map. Its key ingredient is a jointly continuous map that sends each input measure to the uniform distribution on [0, 1].

B.4 PROOF OF THEOREM 3.6

  • The proof truncates the output to a bounded region and covers it with an h-net, constructing a coupling whose transport distance is at most h.The cover uses continuous weights supported near its selected points.Source: §B.4 · pp. 22–23
  • The proof regularizes a discontinuous pushforward map and combines its error with truncation and discretization errors to obtain the approximation bound.The regularized and unregularized maps agree outside a small transition set, up to a Lebesgue-null set.Source: §B.4 · pp. 24–25
  • Uniform integrability of p-th moments on compact Wasserstein families controls the output truncation error as the truncation radius grows.The proof applies this property to the compact image F(K).Source: §B.4 · pp. 22, 25
  • A jointly continuous map H0 sends each input measure in K exactly to Unif[0, 1], using the uniform level set condition.The proof first smooths a scalarization-based cumulative distribution function, then establishes continuity of the resulting map.Source: §B.4 · pp. 26–27

C.1 PROOF OF THEOREM 4.4

The proof of Theorem 4.4 reduces approximation on an unbounded input space to a compact-domain transformer approximation. It embeds and approximates the input, then combines the resulting error bounds with truncation and continuity estimates.

C.1 PROOF OF THEOREM 4.4

  • The bounded injective map ι embeds the input space, and a ReLU MLP approximates ι before subsequent attention layers act on compactly supported measures.The initial identity attention layer and token-wise MLP place the construction in the transformer form used in the proof.Source: §C.1 · pp. 28–30
  • The proof uses the measure-theoretic transformer in-context map approximation theorem on the compact domain P(Ω) × Ω.A continuous extension first supplies the map to be approximated there.Source: §C.1 · pp. 28–29
  • The proof controls the approximation error by coupling the relevant pushforwards and combining bounds for compactification, truncation, and transformer approximation.A common input sample couples the target and constructed outputs.Source: §C.1 · pp. 28, 30

C.2 PROOF OF COROLLARY 4.6

The proof of Corollary 4.6 adapts Theorem 3.6 by replacing the input measure with a continuously varying source measure. Under assumption (4.3), it constructs a continuous map that sends each source measure to Unif[0, 1].

C.2 PROOF OF COROLLARY 4.6

  • Assumption (4.3) and continuity of η yield a continuous map whose pushforward of η(µ) is Unif[0, 1] for every µ in K.The proof obtains the map by the argument used for the continuous exact uniformizer.Source: §C.2 · p. 31
  • The remainder of the proof follows Theorem 3.6 with the uniformized source measure replacing the uniformized input measure.The substitution is H0(µ, · )#µ replaced by eH0(µ, · )#η(µ).Source: §C.2 · p. 31

Related papers

Loading 2609.35483v1…