Source-linked AI summary
Self-conditioned Flow Map Language Models via Fixed-point Flows
Jaehoon Yoo, Wonjung Kim, Floor Eijkelboom, Chanhyuk Lee, Nicholas M. Boffi, Seunghoon Hong, Jinwoo Kim
TL;DR
Self-conditioning improves flow language models, but why it helps and how to use it when distilling flow maps remain unclear. The paper interprets self-conditioning as fixed-point iteration and introduces fixed-point flows, then distills both iterations and flow. Its resulting model achieves state-of-the-art one- and few-step generation on OpenWebText.
Problem
Self-conditioning improves flow language models, but its mechanism is poorly understood, and its timestep dependencies leave its use in flow-map distillation unclear.
Source: §1 · p. 1Method
The paper formalizes self-conditioning as fixed-point iteration in fixed-point flows and distills both the iterations and the flow process.
Source: §1 · pp. 1–2Results
FMLM⋆ achieves state-of-the-art one- and few-step generation on OpenWebText under matched sampling budgets.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
Under contractivity, the fixed-point iteration converges, and additional iterations empirically improve generation quality.
Source: §1 · p. 2, §6 · p. 10Convergence analysis assumes contractivity, which is not guaranteed for a learned denoiser.
Source: §3.2 · p. 5
Abstract
from arXiv · showhide
Self-conditioning is a core technique that enhances continuous flow-based language models, where the model learns to denoise generated text by conditioning on its own denoising estimate. While empirically successful, its performance improvements are poorly understood. Moreover, there is growing interest in the use of few-step generators based on flow maps, for which how to leverage self-conditioning is unclear. Here, we show that flow language models with self-conditioning perform a fixed-point iteration that improves generation through iterative refinement. We use this viewpoint to formulate fixed-point flows, a two-dimensional class of self-conditioned flows, where the first dimension represents the flow process and the second represents the fixed-point iteration. We show that fixed-point flows define valid flow maps, and show that they can be distilled from self-conditioned flow models by compressing both fixed-point iterations and the flow process, the former with fixed-point distillation and the latter with flow map distillation. Our resulting flow map language model, FMLM$^\star$, outperforms state-of-the-art self-conditioned models and few-step models in one- and few-step generation on OpenWebText. Code is available at https://github.com/Ugness/self-conditioned-fmlm.
1 INTRODUCTION
Self-conditioning improves flow language models, but its mechanism and use in flow-map distillation are not well understood. The paper frames self-conditioning as fixed-point iteration, introduces fixed-point flows, and reports state-of-the-art one- and few-step generation on OpenWebText.
1 INTRODUCTION
- Despite empirical success, why self-conditioning improves flow models is poorly understood, and its dependencies across generation timesteps complicate flow-map distillation.Source: §1 · p. 1
- Fixed-point flows formalize self-conditioned flows with separate dimensions for the original flow and fixed-point iterations, which empirically improve generation quality.Source: §1 · p. 1
- Self-conditioning can be understood as a fixed-point iteration that refines predictions, with convergence theoretically established under a contractivity assumption.Source: §1 · p. 2
- Fixed-point flows define valid flow maps that can be learned by compressing both fixed-point iterations and the flow process.Source: §1 · pp. 1–2
- The distilled models achieve state-of-the-art one- and few-step generation on OpenWebText, and self-conditioned models can be distilled without degradation into models without self-conditioning.Source: §1 · p. 2
2 PRELIMINARY
Flow language models represent text in a continuous embedding space and generate samples through deterministic evolution. Their flow maps transport samples between timesteps and support few-step generation.
2 PRELIMINARY
- Text is embedded in a continuous space, where flow language models learn the induced data distribution before decoding generated embeddings back into tokens.Source: §2 · p. 2
- Flow matching defines a path by interpolating noise and data, then uses a learned velocity field and numerical integration to generate samples.Source: §2 · pp. 2–3
- Language models commonly learn a denoiser that predicts the mean of clean data, from which the flow velocity can be recovered.Source: §2 · p. 3
- A flow map is the solution operator transporting samples between timesteps, allowing sequential few-step generation or one-step transport from noise to data.Source: §2 · p. 3
- Existing flow-map distillation methods learn maps from a flow model's velocity by satisfying mathematical relations between velocity and the map.Source: §2 · p. 3
3 THEORETICAL FRAMEWORK
The framework interprets self-conditioning as iterative refinement toward a fixed point, then uses fixed-point flows and their flow maps to formulate distillation for few-step generation.
3.1 SELF-CONDITIONED FLOW LANGUAGE MODELS
- Self-conditioning leaves the ideal denoising target unchanged, while generation updates both flow and denoising states, coupling estimates across timesteps.In practice, the self-conditioned loss can encourage a self-correcting prediction.Source: §3.1 · p. 4
3.2 SELF-CONDITIONING INDUCES A FIXED-POINT ITERATION
- Under contractivity, repeated denoiser updates converge exponentially to a unique fixed point, with denoising error bounded along the iteration and at convergence.The fixed-point error bound depends on the initial prediction error and the first update's magnitude.Source: §3.2 · p. 5
3.3 FIXED-POINT FLOWS
- Replacing the conditioning state with its fixed point yields an ordinary flow; with a Bayes-optimal fixed-point denoiser, its velocity recovers the true velocity.Fixed-point distillation can produce a self-conditioning-free denoiser, and conventional self-conditioned generation approximates the fixed point with one warm-started iteration.Source: §3.3 · pp. 5–6
- Cold-start sampling with sufficient fixed-point iterations can replace warm starts in ELF and LangFlow.Warm starts use the previous flow timestep's fixed-point estimate when fixed points are locally correlated across time.Source: §3.3 · p. 6
3.4 FIXED-POINT FLOW MAPS
- Fixed-point flows define valid flow maps; two-time denoisers match the fixed-point denoiser on the diagonal and satisfy semigroup consistency off the diagonal, enabling few-step distillation.The two-time denoiser's learning objective enforces these diagonal and semigroup conditions.Source: §3.4 · pp. 6–7
4 EXPERIMENTS
Experiments find that fixed-point refinement improves generation and that self-conditioning can be distilled into a self-conditioning-free model or a few-step flow map. On OpenWebText, offline FMLM⋆ achieves the best gPPL among baselines preserving data-level entropy.
4 EXPERIMENTS
- Experiments use OpenWebText sequences of 1024 tokens and assess generation using gPPL and average per-sample unigram entropy against data entropy of 5.44 nats.The analysis uses 256 samples and distillation uses 1024 samples.Source: §4 · p. 7
4.1 ANALYZING SELF-CONDITIONED FLOW LANGUAGE MODELS
- Across ELF and LangFlow, relative distance to the fixed point decays and gPPL decreases as fixed-point iterations increase, while entropy stays near data entropy.The Figure 2 comparisons use 32 sampling steps for ELF and 64 for LangFlow.Source: §4.1 · p. 8, Figure 2 · p. 8
- With one fixed-point iteration, warm-start outperforms cold-start; with 100 iterations, both reach the same gPPL-entropy frontier.The result is consistent with initialization mattering less once sufficient iterations are run.Source: §4.1 · p. 8, Figure 3 · p. 8
4.2 DISTILLING SELF-CONDITIONING INTO FIXED-POINT FLOW MAPS
- Offline FMLM⋆ achieves the best gPPL among baselines preserving data-level entropy (5.44 nats) on OpenWebText.Online distillation with 8 FPIs achieves competitive quality while reducing training time from 82 to 23 hours.Source: Table 1 · p. 9, §4.2 · p. 9
- ELF⋆ matches ELF’s gPPL-entropy frontier after fixed-point distillation, showing that self-conditioning can be removed without degrading quality.The comparison uses 32-step generation.Source: §4.2 · p. 9, Figure 4 · p. 9
- With 8 generation steps, FMLM⋆ approaches the 32-step frontier of the self-conditioned teacher ELF.The model is distilled from ELF using a fixed-point denoiser and a semigroup distillation objective.Source: §4.2 · p. 9, Figure 5 · p. 9
5 RELATED WORK
Related work connects self-conditioning to prediction updates, fixed-point computation, and flow-map generation. This paper develops the fixed-point perspective to analyze self-conditioning and address its integration into flow maps.
Self-conditioning in continuous language models
- Prior work interprets self-conditioned denoisers as fixed points of prediction updates; this paper analyzes convergence under contractivity and separates refinement from flow evolution to define fixed-point flows.The cited interpretation is attributed to Gulrajani & Hashimoto (2023).Source: §5 · pp. 9–10
Fixed-point computation in neural architectures
- Fixed-point iterations have been used to refine neural representations and generative denoisers, while this work formalizes self-conditioning in flow language models using that perspective.Examples include hidden-state root-finding and fixed-point iterations in diffusion and masked generative denoisers.Source: §5 · p. 10
Flow map language models
- Flow-map models support one- and few-step text generation, but integrating self-conditioning remains challenging because denoisers depend on prediction states that evolve across sampling steps.Prior flow-map approaches include FMLM, CFM, and DFM.Source: §5 · p. 10
6 CONCLUSION
The conclusion frames self-conditioning as fixed-point refinement and uses fixed-point flows to obtain FMLM⋆. It reports OpenWebText performance under matched sampling budgets and identifies teacher dependence and text-only evaluation as boundaries.
6 CONCLUSION
- Under contractivity, the self-conditioned iteration converges to a fixed point, and additional iterations empirically improve generation quality.Conventional self-conditioned sampling is interpreted as a single-step warm-started fixed-point iteration.Source: §6 · p. 10
- FMLM⋆ is distilled by compressing fixed-point iterations and flow, achieving state-of-the-art one- and few-step generation on OpenWebText under matched sampling budgets.Source: §6 · p. 10
- Flow-map learning requires a separate teacher, and evaluation is limited to text, leaving applicability to other modalities open.Source: §6 · p. 10
A.2 PROOF OF APPROXIMATE CONTRACTIVITY
The appendix argues that small self-conditioned loss yields an approximately contractive region with high probability and limited leakage under the training distribution. The result is restricted to pairs separated by at least a specified scale.
A.2 PROOF OF APPROXIMATE CONTRACTIVITY
- A sufficiently small self-conditioned loss yields a closed, high-probability set that is contractive above scale w and approximately forward-invariant with leakage at most ε.The passage connects the loss terms to set probability and to keeping f(Z) within the set for most Z sampled from μ.Source: §A.2 · p. 14
- The set A is closed, and its image under f remains in O with the stated approximate forward-invariance guarantee.Source: §A.2 · p. 14
- Approximate contractivity is established for pairs z, z′ in A whose separation is at least w.The conclusion passage states the pairwise scale condition; the preceding lemma names the result approximate contractivity.Source: §A.2 · p. 15
A.3 PROOF OF PROPOSITION 3.3
Under a contraction assumption, repeated self-conditioned denoising converges to a unique fixed point, with bounds on the denoising error during iteration and at convergence.
A.3 PROOF OF PROPOSITION 3.3
- The iteration converges exponentially to a unique fixed point from any initialization in the contraction domain.The contraction factor satisfies 0 ≤ η < 1.Source: §A.3 · pp. 15–17
- At any iteration j ≥ 1, the denoising error is bounded under the contraction assumption.The supplied proposition states this bound but does not include its full expression.Source: §A.3 · p. 17
- At convergence, denoising error is bounded by the initial denoising error plus a contraction-scaled first-step increment.The bound adds η/(1−η) times |z1−z0| to the initial denoising error.Source: §A.3 · p. 17
A.4 PROOF OF PROPOSITION 3.4
These propositions connect self-conditioned fixed points to ordinary flow dynamics, give iteration and flow-map properties, and characterize consistency conditions for a two-time denoiser.
A.4 PROOF OF PROPOSITION 3.4
- The fixed-point denoiser depends only on time and input, yielding an ordinary time-dependent ODE; if it is Bayes-optimal, its velocity recovers the true velocity.Source: §A.4 · pp. 17–18
A.5 ITERATION BOUND UNDER CONTRACTIVITY
- Under contractivity, an iteration count guarantees error at most ε; no iterations suffice when the initial error is within ε, and one suffices when η = 0.Source: §A.5 · p. 18
A.6 PROOF OF PROPOSITION 3.5
- Where the fixed-point ODE has unique solutions, it defines a flow map with the composition law across intermediate times.Evolving from s to t is equivalent to evolving first from s to u and then from u to t.Source: §A.6 · p. 18
A.7 PROOF OF PROPOSITION 3.6
- The two-time denoiser recovers the flow map, and its semigroup condition is equivalent to the stated identity involving an intermediate time.Source: §A.7 · pp. 19–20
A.8 PROOF OF PROPOSITION 3.7
- The true two-time denoiser has zero population loss, and any zero-loss solution satisfies the diagonal and semigroup conditions almost surely under the training distribution.Source: §A.8 · p. 20
B.1 ANALYSIS DETAILS
The appendix describes the convergence and generation analyses, implementation details for the distilled models, and the observed quality trend across fixed-point iterations.
B.1 ANALYSIS DETAILS
- Fixed-point convergence was assessed over 256 training examples and ten time points, using 200 damped Picard iterations with α = 0.3.The run starts at z0 = 0, and its final iterate serves as the reference fixed point.Source: §B.1 · p. 21
B.2 SELF-CONDITIONING-FREE MODEL DETAILS
- ELF⋆ is distilled from a frozen self-conditioned teacher using consistency deep equilibrium distillation and generates with one evaluation per flow timestep.At inference, the self-conditioning input is set to zero.Source: §B.2 · p. 21
B.3 SELF-CONDITIONED FLOW MAP DETAILS
- FMLM⋆ trains a two-time denoiser with diagonal and off-diagonal examples, using separate diagonal and semigroup losses with equal weights.The diagonal-example probability is 0.4 for offline distillation and 0.5 for online distillation.Source: §B.3 · p. 22
C.1 GENERATION QUALITY ACROSS FIXED-POINT ITERATIONS
- As fixed-point iterations increase, gPPL decreases for ELF and LangFlow while entropy remains near the data entropy of 5.44 nats.Source: Table 2 · p. 22, §C.1 · p. 22
C.2 ONLINE FIXED-POINT DISTILLATION
Online fixed-point distillation gains in gPPL up to eight iterations, while further iterations bring limited or negative returns alongside increased training time.
C.2 ONLINE FIXED-POINT DISTILLATION
- With 16 FPIs, gPPL improves only slightly for one- and two-step generation, worsens for four-step generation, and training time rises from 23 to 32 hours.The reported results suggest diminishing returns from additional FPIs.Source: §C.2 · pp. 22–23, Table 3 · p. 23
D.1 ITERATIVE REFINEMENT AT A FIXED FLOW STATE
At a fixed flow state, fixed-point iterations improve grammatical consistency and contextual coherence; the section also provides FMLM⋆ samples at one, two, and four sampling steps.
D.1 ITERATIVE REFINEMENT AT A FIXED FLOW STATE
- The analysis monitors ELF predictions over three fixed-point iterations at t = 0.25 using an interpolant from a training sequence and Gaussian noise.Source: §D.1 · p. 23
- Fixed-point iterations improve grammatical consistency and contextual coherence, while gPPL decreases and entropy stays close to the data distribution.The qualitative changes include more grammatical phrasing and wording that fits references to patients and counselors.Source: §D.1 · p. 23, Figure 6 · p. 23
D.2 FEW-STEP GENERATION SAMPLES
- FMLM⋆ generation samples are shown for one-, two-, and four-step sampling.Source: §D.2 · p. 24, Figure 7 · p. 25, Figure 8 · p. 26