PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Flow Language Models

Arnav Chavan, Chanhyuk Lee, Udbhav Bamba, Jaehoon Yoo, Seunghoon Hong, Aditi Raghunathan, Jinwoo Kim

arXiv:2610.28559v1cs.CLcs.AI

TL;DR

Effective techniques for building flow language models at billion-parameter scale remain less established. Lafa adapts pretrained AR models into a blockwise flow language model, and its 7B model achieves 2.54× the throughput of a 7B AR SFT baseline at comparable average accuracy.

  • Problem

    Effective techniques for building flow language models at billion-parameter scale remain less established, despite their potential for parallel generation.

    Source: §1 · pp. 1–2

  • Method

    Lafa adapts pretrained AR models into a blockwise flow model with asynchronous flows within blocks and AR prediction on the prefix.

    Source: §1 · p. 2

  • Results

    2.54× throughput over the 7B AR SFT baseline at comparable average accuracy; across instruction following, reasoning, and coding, Lafa matches or surpasses similarly sized AR, MTP, and discrete-diffusion models.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results demonstrate a practical path to scaling continuous flows for language modeling through data-efficient AR adaptation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Continuous flow language models offer a path to parallel generation with tools for control and acceleration unavailable to discrete diffusion, yet effective techniques for building them at the billion-parameter scale remain less established. We present Lafa (Large flow language model), a fast, high-performing flow language model scaled to 8B parameters in a data-efficient manner. Lafa is a blockwise flow model with causal attention on the prefix and bidirectional, asynchronous flow within each block. This design enables efficient training by adapting pretrained autoregressive (AR) models and fast inference via prefix caching and blockwise parallel decoding. We train Lafa at 0.5B, 7B, and 8B scales with fewer than 30B tokens, more efficient than previous diffusion models, and evaluate it on instruction following, reasoning, and coding, where the learned flow matches or surpasses the quality of AR and can be distilled into flow maps for one- and few-step generation. Notably, flow training preserves or even improves pretrained AR capabilities via a multi-token prediction effect, which we leverage in a self-speculative decoding scheme that verifies flow-based drafts autoregressively within a single model. This yields 2.54x the throughput of an AR model at comparable quality, outperforming multi-token prediction and discrete diffusion methods. Our findings show that continuous flows can serve as a scalable foundation for language modeling.

1 Introduction

Effective billion-parameter flow language models remain less established, despite continuous flows’ potential for parallel generation. Lafa scales blockwise flow training to 8B parameters and reports a quality–speed frontier that surpasses similarly sized AR, MTP, and discrete-diffusion models.

1 Introduction

  • Continuous flows support parallel generation and can use control and acceleration techniques developed for image and video generation that are unavailable to discrete methods.The paper identifies limited demonstrations at small scales and on simple tasks, alongside training inefficiency in a prior scaling attempt, as gaps in established scaling techniques.Source: §1 · pp. 1–2
  • 2.54× throughput over a strong AR baseline at matched quality, with Lafa 7B outperforming existing parallel-generation methods.Figure 1 reports 2.02–2.54× throughput at matched or better quality; the 2.54× comparison is against a strong AR baseline.Source: Figure 1 · p. 1, §1 · p. 2
  • Lafa is a blockwise flow model with causal attention on the prefix and bidirectional, left-to-right-biased asynchronous flow within each block.The design enables training by adapting pretrained AR models and inference through prefix caching and parallel flow decoding.Source: §1 · p. 2
  • Flow training preserves or enhances pretrained AR capabilities through a multi-token prediction effect, which Lafa uses for self-speculative decoding.The scheme verifies flow-based drafts autoregressively.Source: §1 · p. 2
  • Lafa’s learned flow matches or surpasses similarly sized AR, MTP, and discrete-diffusion models across instruction following, reasoning, and coding.The paper reports scaling Lafa to 7–8B parameters with fewer than 30B tokens.Source: §1 · p. 2

2 Large flow language models

Lafa builds blockwise asynchronous flows on embedded tokens, adapting pretrained autoregressive models for parallel training and inference. Its design combines prefix-causal attention with bidirectional attention and independently learned progress schedules within each block.

2.1 Block asynchronous flows

  • Tokenwise time lets earlier block positions progress faster, while a denoiser predicts clean tokens and its learned velocity transports noisy embeddings toward them.The schedules are learned from denoising loss; the resulting flow is generated by integrating an ODE.Source: §2.1 · p. 3

2.2 Learning the flow from an autoregressive model

  • Lafa adapts a pretrained AR model into a block denoiser using time conditioning, block-bidirectional attention, constrained embedding updates, and joint AR and denoising training.Causal prefix keys and values remain independent of block state, enabling reuse across denoising steps and subsequent blocks.Source: §2.2 · p. 4
  • One forward pass trains all blocks: clean positions use causal AR predictions, while noisy blocks attend to preceding clean tokens and bidirectionally within themselves.The training sequence pairs clean data with a copy divided into noisy blocks.Source: §2.2 · p. 4, Figure 4 · p. 5
  • Loss-derived tokenwise schedules slow progress at difficult noise levels and generally place earlier positions ahead; joint AR supervision further improves flow accuracy and AR accuracy over matched AR SFT.The joint objective combines denoising cross entropy and embedding squared error, while learned embedding directions retain pretrained norms.Source: §2.2 · pp. 4–5

Flow and flow map decoding.

Self-speculative decoding.

  • Self-speculative decoding drafts a block with one flow step, then commits the longest prefix agreeing with greedy AR predictions, preserving the retained model’s greedy output.Drafting and verification can occur in a single forward pass by preparing candidates for possible accepted-prefix lengths.Source: Self-speculative decoding · p. 6

3 Experiments

The experiments evaluate Lafa across model scales and benchmarks, testing flow generation, retained autoregressive accuracy, self-speculative decoding, and design ablations. Results cover six benchmarks for single-token and self-speculative evaluation, and three for flow decoding, throughput, and ablations.

3.1 Flow and flow map decoding

  • Across scales, flow accuracy generally rises with more integration steps; block-size-four decoding approaches AR SFT at 0.5B and surpasses it at 7B.Flow-map distillation improves one- and two-step accuracy; at 7B, its two-step student approaches the four-step teacher, though a gap remains versus flow decoding with more integration steps.Source: §3.1 · p. 7, Figure 6 · p. 7

3.2 Autoregressive decoding

  • Joint flow training improves average single-token accuracy over matched AR SFT at all three scales, and at 8B matches Efficient-DLM with at least 18× fewer training tokens.At 0.5B and 7B, Lafa also outperforms matched multi-token and block-parallel baselines on average.Source: Table 1 · p. 7, §3.2 · p. 8

3.3 Self-speculative decoding

  • Across scales, Lafa reaches approximately 2.5–5 TPF while remaining close to or above AR SFT accuracy.At 7B, block size sixteen nearly doubles TPF over block size four while keeping accuracy close to AR SFT; at 8B, Lafa has similar accuracy and TPF to Efficient-DLM despite at least 18× fewer training tokens.Source: Figure 7 · p. 8, Table 2 · p. 8, §3.3 · p. 9
  • At 7B, block-size-sixteen self-speculative decoding reaches 2.54× the throughput of 7B AR SFT at comparable average accuracy.Throughput was measured in tokens per second on one H100 GPU with batch size one, averaged over IFEval, GSM8K, and HumanEval.Source: §3.3 · p. 9, Figure 8 · p. 9

3.4 Ablations

  • Block-bidirectional attention, learned tokenwise time, constrained embedding-direction updates, combined denoising and embedding objectives, and prefix AR loss each improve flow accuracy over their stated alternatives.Self-conditioning gives the larger inference gain and requires no additional model evaluations during decoding.Source: §3.4 · p. 9, Table 3 · p. 9

4 Related work

Related work covers block-autoregressive, discrete diffusion, and continuous diffusion or flow approaches to parallel language modeling. These methods differ in how they generate blocks or sequences and in the challenges they face at few-step generation or practical scale.

Block-autoregressive language models

  • Block-autoregressive models generate blocks conditioned on preceding blocks, using MTP or masked/noisy-block predictors; independent within-update prediction can degrade quality when many tokens are committed.Some approaches use AR verification or confidence-based acceptance, while learned deterministic noise-to-token maps can capture joint dependencies without independent output sampling.Source: §4 · p. 10

Discrete diffusion language models

  • Discrete diffusion reverses a discrete corruption process through iterative denoising, but independent sampling within updates can limit few-step quality.Block diffusion combines iterative denoising within each block with autoregressive generation and prefix caching across blocks.Source: §4 · p. 10

Continuous diffusion and flow language models

  • Continuous diffusion and flow generate text through noisy continuous representations; flow models transport joint sequence representations and can learn flow maps for few-step generation.Lafa aligns blockwise asynchronous flows with pretrained autoregressive structure to support efficient adaptation, prefix caching, and self-speculative generation.Source: §4 · p. 10

5 Conclusion

Lafa adapts pretrained autoregressive models into an 8B blockwise flow model using fewer than 30B training tokens. Its retained autoregressive predictor verifies one-step flow drafts, and Lafa 7B achieves 2.54× the throughput of 7B AR SFT at comparable average accuracy.

5 Conclusion

  • Lafa reaches 8B parameters by adapting pretrained autoregressive models with fewer than 30B training tokens.It combines causal generation across blocks with learned asynchronous flows within each block, preserving prefix caching while enabling parallel generation.Source: §5 · p. 10
  • With sufficient integration steps, flow decoding approaches or surpasses AR SFT accuracy, while flow-map distillation improves few-step generation.Joint training also improves AR accuracy over matched AR SFT; the retained AR predictor verifies one-step flow drafts within the same model.Source: §5 · p. 10
  • Lafa 7B achieves 2.54× the throughput of 7B AR SFT at comparable average accuracy; improving flow-map distillation remains a direction for further acceleration.The conclusion identifies better incorporation of self-conditioning and prefix guidance as a promising direction.Source: §5 · p. 10

A Theoretical results

The appendix derives a relationship between the asynchronous flow’s velocity field and its denoiser. The derivation identifies a conditional-mean embedding as part of that relationship.

A.1 Asynchronous flow velocity and denoiser

  • The appendix derives the relationship between the velocity field and denoiser of the asynchronous flow introduced in Section 2.1.The derivation begins from the time derivative of the interpolant and takes its conditional expectation to obtain the velocity.Source: §A.1 · p. 17
  • Proposition A.1 states a position-wise velocity relationship for the interpolant, with the velocity defined as the conditional expectation of its time derivative.Source: §A.1 · p. 17
  • The derivation solves the interpolant for the noise and substitutes that expression into the conditional velocity.The resulting conditional mean is identified as the posterior mean embedding under the stated distribution.Source: §A.1 · p. 17

A.2 Asynchronous embedded flow maps

The appendix characterizes two-time denoisers whose induced flow maps match the asynchronous flow, then gives a KL-based distillation objective. Under full-support sampling, every population minimizer recovers the teacher flow map.

A.2 Asynchronous embedded flow maps

  • A two-time denoiser recovers the true flow map when it satisfies diagonal and semigroup conditions, with the semigroup condition composing transitions through an intermediate time.The diagonal condition makes the denoiser approach the frozen teacher denoiser as the two times coincide.Source: §A.2 · p. 18
  • The distillation objective uses KL losses to enforce diagonal matching and semigroup consistency, with the student initialized from the frozen teacher.The off-diagonal target composes shorter student transitions, and the objective combines flow-map distillation with autoregressive distillation.Source: §A.2 · p. 19
  • The population objective attains zero, and every minimizer recovers the teacher flow map through the two-time denoiser construction, assuming full-support sampling.The result is stated for the population objective over two-time denoisers.Source: §A.2 · p. 19

B Algorithmic details

  • The appendix states that training and inference pseudocode appears in Algorithms 1–4 and that main-result hyperparameters are listed in Table 4.Source: B Algorithmic details · p. 19

B.1 Training

Training uses a prefix-causal, block-bidirectional attention mask and learned tokenwise time schedules; inference applies flow decoding with self-conditioning and prefix guidance.

B.1 Training

  • Noisy blocks attend bidirectionally within each block and to preceding clean tokens, while clean positions remain causal and exclude later clean tokens and other noisy blocks.The attention structure is applied to every block at once during training.Source: §B.1 · p. 19
  • Tokenwise times are learned as inverse CDFs of position-specific denoising losses, allocating more training samples and integration steps to difficult noise levels.The loss estimates are maintained in an exponential moving average over uniform time bins; independent uniform estimates prevent low-probability regions from being starved.Source: §B.1 · p. 20
  • Training can self-condition by using an initial no-gradient prediction’s expected token embeddings to condition a second pass, with probability 0.5 per batch in practice.The second-pass training loss is computed on the self-conditioned input.Source: §B.2 · p. 20

B.2 Flow decoding

  • At inference, the previous denoiser prediction supplies self-conditioning, while prefix guidance modifies the prediction used in Euler integration.Guidance scale g = 1 recovers the usual prediction; g > 1 changes the velocity and therefore the sampled distribution.Source: §B.2 · p. 21

B.3 Self-speculative decoding

Fused self-speculative decoding combines autoregressive verification of a previous flow draft with drafting the next block in one forward pass. Candidate drafts correspond to possible accepted prefixes, and the verified prefix selects which draft to use.

B.3 Self-speculative decoding

  • One forward pass verifies a previous draft with the autoregressive predictor while drafting the next block, combining both operations in fused self-speculative decoding.The approach uses the denoiser for drafts and the retained autoregressive predictor for verification.Source: §B.3 · p. 21, Figure 9 · p. 22
  • For each possible accepted prefix, the fused pass produces a candidate next-block draft; verification selects the candidate conditioned on the accepted prefix.The figure illustrates this parallel candidate drafting for block size three.Source: §B.3 · p. 21, Figure 9 · p. 22
  • The procedure reuses the prefix key-value cache to improve efficiency.Source: §B.3 · p. 23

C Evaluation details

The section introduces the evaluation setup and metrics, then reports detailed per-benchmark results.

C Evaluation details

C.1 TPF evaluation and per-benchmark scores

This section defines the forward-pass accounting used for tokens-per-forward and identifies the per-benchmark tables underlying the decoding evaluations.

C.1 TPF evaluation and per-benchmark scores

  • Table 7 reports self-speculative and parallel decoding accuracy and TPF across model scales.The compared verification procedures differ by method, including greedy AR verification for causal block predictors and native or deterministic trajectory verifiers for other methods.Source: Table 7 · p. 24, §C.1 · p. 25
  • Table 6 reports single-token decoding results, while Table 8 gives per-task accuracy and throughput for 7–8B models.Table 8 measurements use one H100 GPU with batch size one; Efficient-DLM is 8B and the other methods are 7B.Source: Table 6 · p. 24, Table 8 · p. 25, §C.1 · p. 25
  • TPF counts committed tokens against forward passes, with the counted passes varying by decoding method.For Lafa and Efficient-DLM, the count includes fused verification-and-redraft passes plus initial drafting; Efficient-DLM prompt prefill and cache refreshes are excluded, while Fast-dLLM v2 includes denoising, prefill, and cache updates.Source: §C.1 · pp. 23, 25

C.2 Decoding throughput

Across confidence thresholds, Lafa’s accuracy compares favorably with AR SFT and, at 8B, remains on par with Efficient-DLM. The section also describes end-to-end throughput measurement and Lafa’s self-speculative decoding setup.

C.2 Decoding throughput

  • Figure 8 measures end-to-end generation throughput in tokens per second on one H100 GPU, with batch size one, bfloat16 inference, and a 32-token warm-up.TPS is total generated tokens divided by total generation time.Source: §C.2 · p. 25
  • The throughput comparison includes AR SFT, DeepSeek-MTP, MARS, Fast-dLLM v2, Efficient-DLM, and Dream, using their stated native or method-specific decoding configurations.The figure caption specifies Lafa’s block sizes and model scales alongside the scales for Efficient-DLM, LLaDA, and other baselines.Source: §C.2 · pp. 25–26
  • Lafa uses a one-step denoiser to draft each next block, then verifies it against its own greedy AR predictions in a fused pass.After selecting the longest verified prefix, it retains the corresponding KV states and uses the matching successor block as the next draft, avoiding a separate redrafting pass.Source: §C.2 · pp. 25–26

C.3 Confidence decoding

  • Confidence decoding accepts the longest left-to-right prefix whose token confidences exceed threshold λ, and requires only the successor block for the known prefix length.The evaluation compares accuracy–TPF tradeoffs across thresholds; AR verification avoids lower-threshold degradation without threshold tuning, so the authors use self-speculative decoding by default.Source: §C.3 · pp. 26–27
  • At 0.5B, Lafa with block size four reaches the highest peak accuracy; at 7B it exceeds AR SFT at high thresholds, and at 8B it stays above AR SFT and on par with Efficient-DLM.The comparison is an accuracy–TPF tradeoff across confidence thresholds; MARS degrades more gracefully as thresholds are lowered.Source: Figure 10 · p. 26, §C.3 · p. 27

Related papers

Loading 2610.28559v1…