Source-linked AI summary
Flow Duality and Source Geometry for Categorical Generation
Etrit Haxholli
TL;DR
The paper addresses whether continuous convex-interpolant flows over one-hot targets induce discrete flow-matching paths after argmax projection. It proves this duality under lifted-coupling, source-symmetry, and regularity assumptions, and derives how Gaussian, bounded-uniform, and centered negative-exponential sources shape the induced coefficient. The resulting formulas make source geometry a structural design choice, while preliminary diagnostics and a short pilot suggest corresponding effects in learned paths and early generative behavior.
Problem
The paper asks whether continuous convex-interpolant flows over one-hot representations induce discrete convex-interpolant flows after position-wise argmax projection.
Method
It analyzes an argmax-compatible lifted coupling between continuous sources and one-hot targets, then derives projected paths and coefficients for Gaussian, bounded-uniform, and centered negative-exponential sources.
Results
The duality holds under the stated symmetry and regularity conditions, with Gaussian timing delayed by vocabulary size, bounded-uniform timing controlled by support width, and centered negative-exponential timing independent of vocabulary size.
Takeaways & Limitations
Continuous source design is a structural part of categorical flow modeling because it determines the effective discrete interpolation path and can influence learned transport geometry.
Takeaways & Limitations
The generative check is a single early-training operating-point diagnostic, and the reported sample-entropy estimate is a unigram-entropy surrogate rather than a full generative frontier.
Abstract
from arXiv · showhide
Continuous and discrete flow matching are usually treated as separate constructions. This paper identifies a duality between them: projecting continuous convex-interpolant paths with one-hot targets through a position-wise argmax yields discrete convex-interpolant paths. The result requires source laws with appropriate coordinate symmetry and boundary regularity, and it makes the continuous source distribution an explicit design choice for categorical generation. We derive the induced discrete interpolation behavior for Gaussian, bounded-uniform, and centered negative-exponential sources, showing that different source geometries lead to qualitatively different transition timing and vocabulary-size dependence. Small visual diagnostics and a short language-modeling pilot suggest that these source-design effects can also appear in learned transports and early generative quality.
1 Introduction
The paper asks whether continuous convex-interpolant flows over one-hot targets induce discrete flow-matching paths after position-wise argmax projection. It proves this duality under symmetry and regularity conditions, then shows that source geometry controls interpolation timing and may affect learned trajectories.
- Duality result: Under an argmax-compatible lifted coupling and suitable symmetric, boundary-regular continuous sources, position-wise argmax projection yields discrete convex-interpolant paths with uniform source.Identical per-position source laws produce a shared interpolation coefficient; otherwise coefficients may depend on position.
- Source geometry: The continuous source is a design choice because Gaussian noise must compete with the maximum of many coordinates, delaying the induced discrete coefficient as vocabulary size grows.Alternative source geometries can change both transition timing and transport geometry.
- Source geometry: The induced coefficient is vocabulary-dependent and delayed for Gaussian sources, support-width-controlled for bounded-uniform sources at fixed vocabulary size, and vocabulary-independent for centered negative-exponential sources.These formulas establish qualitatively different interpolation behavior across source families.
- Empirical motivation: Toy visualizations show that source family, placement, and scale can alter induced coefficients and learned continuous path geometry, but are not large-scale empirical evidence.The diagnostics compare Gaussian and shifted-uniform sources in a two-point target problem.
2 Background and Notation
The background defines discrete flow matching as a probability flow from source to data distributions and continuous flows as pushforwards of convex-interpolant trajectories between coupled endpoints. It also introduces argmax-projected continuous representations and flow-map denoisers relevant to the paper’s duality.
- Discrete Flow Matching: Discrete flow matching transforms a source distribution into a data distribution through time-dependent probability velocities over finite vocabulary sequences.The velocity specifies position-wise transition rates and can be approximated with a neural network for generation.
- Conditional Probability Flows: Discrete conditional paths use an increasing coefficient k_t to interpolate between endpoint states independently at each sequence position.The coefficient satisfies k_0 = 0 and k_1 = 1, with absolute continuity assumed for the corresponding velocity derivation.
- Discrete Flow Matching: Common discrete sources include an all-mask state and the uniform distribution over vocabulary sequences.The mask construction uses an enlarged vocabulary unless the mask token is already included.
- Continuous Flows: Continuous flows are pushforwards of coupled endpoint laws whose conditional trajectories are straight line segments, possibly reparameterized by a nondecreasing coefficient ˜k_t.The continuous source and target are marginals of a probability measure, allowing one-hot-supported targets without requiring a Lebesgue density.
- Related continuous constructions: Argmax projection maps Euclidean latent coordinates to categorical states, while continuous flow maps and two-time denoisers describe transport between times for one-hot targets.At equal times the two-time denoiser reduces to the endpoint denoiser, and at t = 1 it gives the final clean prediction map.
3 Duality
The paper proves that position-wise argmax projection of continuous convex-interpolant paths can produce discrete convex-interpolant paths with a uniform source under symmetry, regularity, and coupling conditions. The induced coefficient depends on the continuous source, producing different vocabulary and transition behavior across source families.
- Flow duality: Position-wise argmax projection yields discrete convex-interpolant conditionals and a uniform discrete source under coordinate-permutation symmetry, tie avoidance, gap regularity, and an argmax-compatible lifted coupling.The projected path also has per-position factorization and the prescribed terminal marginal when the lifted kernel has the data marginal.
- Flow duality: The lifted construction samples a continuous source, maps it to a discrete source with argmax, samples the target through a discrete coupling, and linearly interpolates with its one-hot endpoint.Conditional independence of the target from the continuous source given the source argmax preserves factorization across positions.
- Induced discrete paths: The projected conditional path matches the standard discrete convex-interpolant form, with one shared coefficient in the common-source setting.The coefficient describes whether the target coordinate has overtaken the source coordinate by time t.
- Source-dependent coefficients: Gaussian sources delay the induced transition as vocabulary size grows because the target coordinate must overtake the maximum of more noise coordinates.The vocabulary dependence is explicit in the Gaussian coefficient.
- Source-dependent coefficients: Bounded-uniform coefficients depend on support width and can reach 1 before the endpoint, whereas centered negative-exponential coefficients have no explicit vocabulary-size dependence.Uniform interval location changes continuous trajectory geometry but not the induced coefficient for fixed width and vocabulary size.
- Generative check: A tiny early-training pilot found a 0.380-nat, or 19.9%, estimated-KL reduction for centered negative-exponential versus Gaussian sources, with only a modest sample-entropy decrease.The estimate is a single operating-point diagnostic and the result is preliminary.
4 Conclusion
The paper establishes a flow duality: under lifted-coupling and source-symmetry assumptions, argmax projection converts continuous convex-interpolant flows into discrete conditional paths. Source geometry determines the effective discrete path, motivating source design for categorical generation.
- Argmax projection turns continuous convex-interpolant flows with one-hot targets into discrete convex-interpolant conditional paths under the stated lifted-coupling and source assumptions.
- Gaussian sources delay the induced coefficient as vocabulary size grows, whereas bounded-uniform and centered negative-exponential sources have support-width-controlled and vocabulary-independent behavior, respectively.
- The source law therefore determines the effective discrete path, providing a mathematical basis for future source-design work in categorical generation.
A.1 Proof of Lemma 3.1
The proof establishes that the position-wise argmax map is measurable by characterizing each token sequence’s preimage as a finite product of Borel sets.
- Argmax regions for individual coordinates are finite intersections of open or closed half-spaces and are therefore Borel.
- The preimage of any sequence under position-wise argmax is a finite product of these Borel regions.
- Consequently, argmax is measurable from the continuous Euclidean space to the discrete sequence space.
A.2 Proof of Lemma 3.2
The proof uses coordinate-permutation invariance and almost-surely unique maxima to show that argmax produces a uniform categorical source at every position.
- Almost-surely unique maxima and coordinate-permutation invariance make every coordinate equally likely to attain the argmax.
- Because token probabilities sum to one, each vocabulary item has probability 1/V under the argmax pushforward.
- A product source with independent positions therefore yields independent uniform tokens and a uniform distribution on [V]^L.
A.3 Proof of Lemma 3.3
The proof shows that the lifted construction preserves position-wise conditional independence: conditioning on discrete endpoints leaves continuous source blocks independent across positions.
- The product source makes initial coordinate blocks independent, with each position’s discrete source token determined only by its own block.
- Under the lifted coupling, the target endpoint is conditionally independent of the continuous source given the discrete source, so conditioning on both endpoints preserves source-block independence.
- The one-hot target is deterministic after conditioning on the target sequence, so endpoint pairs remain independent across positions.
- Since each interpolated block depends only on its corresponding endpoint pair, the conditional path factorizes across positions.
A.4 Proof of Theorem 3.4
The proof shows that argmax projection of the lifted continuous convex interpolants produces a discrete transition whose coefficient is shared across all distinct token pairs. Coordinate-permutation symmetry establishes this pair-independence, while tie and boundary regularity handle the transition events.
- A.4 Proof of Theorem 3.4: The transition probability is expressed through the source gap between the source-argmax and target coordinates and the ratio c_t = k̃_t/(1 − k̃_t).The target wins when its boosted coordinate overtakes the source maximum, apart from equality events handled by the no-atom assumption.
- A.4 Proof of Theorem 3.4: The projected path can switch only between the source argmax token and the one-hot target token.When the source and target tokens differ, the target coordinate is boosted along the convex interpolant, while the largest non-target coordinate remains the source argmax.
- A.4 Proof of Theorem 3.4: Equal source and target tokens yield coincident point masses, so the coefficient value is immaterial in that degenerate case.The proof defines the same coefficient by convention for notational consistency.
- A.4 Proof of Theorem 3.4: Coordinate-permutation invariance makes the transition coefficient identical for every distinct source-target token pair.The permutation maps any ordered distinct pair to any other while preserving the relevant numerator and denominator probabilities.
B Source coefficient derivations
The source-coefficient derivations analyze Gaussian and bounded-uniform sources through target-win probabilities under the convex-interpolant ratio. They show that Gaussian behavior depends on vocabulary size, whereas bounded-uniform behavior depends on support width.
- Gaussian source: The Gaussian coefficient is derived by conditioning on a source coordinate being the argmax and computing when the target coordinate overtakes it.The remaining source coordinates become truncated normals, and the target wins when it lies within c_t of the source maximum.
- Bounded-uniform source: For a bounded-uniform source, coordinate comparisons depend on the interval width rather than its location.The common shift cancels from all comparisons, while the target-win probability is computed from the bounded support.
B.3 Centered negative-exponential source
For the centered negative-exponential source, centering changes the source location without changing argmax comparisons. The resulting coefficient is independent of vocabulary size.
- B.3 Centered negative-exponential source: Centering the negative-exponential source preserves argmax comparisons, so the target-win calculation depends on exponential gaps rather than absolute location.If the target gap is below c_t, the target wins with probability one; otherwise the bounded threshold is evaluated on the exponential tail.
- B.3 Centered negative-exponential source: The centered negative-exponential coefficient is independent of the vocabulary size V.This contrasts with the Gaussian coefficient, whose target competition is delayed as V grows.