PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling

Zijian Jia, Peixuan Zhang, Genying Li, Di Wu, Yujie Fu, Si Li, Boxin Shi, Shuchen Weng

arXiv:2610.32604v1cs.AI

TL;DR

Continuous diffusion has a temporal-compression problem because embedding geometry and token-identity progression do not align, while prior work lacks an analytical account of how geometry governs identity decay. The paper develops a theory of pairwise token competition and a retention-based schedule, then introduces LLaCO. Its 8B model outperforms prior continuous diffusion baselines and rivals comparable-scale discrete diffusion models.

  • Problem

    Continuous diffusion trajectories misalign geometrically with discrete vocabulary identities, causing non-uniform identity decay that prior methods address with heuristics rather than an analytical account of representation geometry.

    Source: §2.3 · p. 3

  • Method

    The paper formalizes pairwise token overtaking with Competition Threshold Theory, proposes Semantic Retention Scheduling, and builds LLaCO with spherical geodesic trajectories and a directional adapter.

    Source: §1 · p. 2

  • Results

    LLaCO’s 8B model outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The theory links token-identity metrics to statistical aggregations of pairwise crossings and motivates allocating denoising time according to logarithmic rank decay.

    Source: §1 · p. 2

    Theoretical conclusions rely on near-orthogonal structures and statistical approximations in high-dimensional, sparsely populated representation spaces; applicability to other geometries and noise distributions needs validation.

    Source: Limitations · p. 13

Abstract

from arXiv · show

Continuous diffusion models have emerged as a promising non-autoregressive paradigm for text generation over continuous embedding spaces. Along the continuous trajectory, the residual token identity directly shapes the denoising objective. However, the geometric trajectory misaligns with temporal progression, triggering severe temporal compression. In this paper, we show that embedding norm decay causes Euclidean distance to yield spurious overtaking, indicating that token competition must be evaluated directionally. In this space, a candidate overtakes the source token only when the effective noise magnitude surpasses their semantic separation. We formalize these pairwise dynamics as the Competition Threshold Theory (CTT). Crucially, high-dimensional overtaking requires extreme noise levels, causing standard trajectories to concentrate identity decay into a narrow window near the noisy endpoint. To resolve this, we propose Semantic Retention Scheduling (SRS) to align time allocation with the progression of identity decay, and demonstrate that existing token identity metrics are fundamentally statistical aggregations of these pairwise crossings. Building on these theoretical insights, we introduce LLaCO, a generative framework powered by competition-aware semantic scheduling. LLaCO adapts a pretrained backbone to continuous representations via an embedding directional adapter and timestep-conditioned modulation. To operate within the directional space, LLaCO employs a spherical noising process formulated as a geodesic interpolation toward a mask-biased endpoint, eliminating norm decay while preserving accurate token competition. Guided by SRS, LLaCO reallocates the denoising trajectory proportionally to the logarithmic rank decay. Instantiated as an 8B model, LLaCO outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.

1 Introduction

The paper analyzes why token identity decays unevenly along continuous diffusion trajectories and develops theory and scheduling to address that mismatch. It introduces LLaCO, whose 8B instantiation outperforms prior continuous diffusion baselines and rivals comparable-scale discrete diffusion models.

1 Introduction

  • Embedding norm decay makes Euclidean distance produce spurious overtaking, while directional competition occurs only when effective noise exceeds semantic separation.The paper formalizes these pairwise dynamics as Competition Threshold Theory; high-dimensional overtaking requires extreme noise, compressing identity decay near the noisy endpoint.Source: §1 · p. 2
  • Semantic Retention Scheduling reallocates timesteps to align with identity decay and address temporal compression.The schedule follows logarithmic rank decay, dedicating maximum optimization budget to identity transitions.Source: §1 · p. 2
  • Existing token identity metrics are statistical aggregations of pairwise crossings, and pretrained-embedding measurements closely track the theoretical derivations.The paper presents this as empirical validation of its competition analysis.Source: §1 · p. 2
  • LLaCO’s 8B instantiation outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.The introduction reports extensive benchmark results but supplies no individual scores here.Source: §1 · p. 2
  • LLaCO adapts a pretrained backbone with a directional adapter, timestep-conditioned modulation, spherical noising, and trajectory reallocation.Its geodesic interpolation toward a mask-biased endpoint eliminates norm decay while preserving accurate token competition.Source: §1 · p. 2

2 Related Work

Prior language-generation paradigms offer different trade-offs: autoregressive models decode sequentially, discrete diffusion supports global revision, and continuous diffusion permits smooth trajectories but lacks an analytical account of identity decay.

2.1 Autoregressive Language Models

  • Autoregressive generation couples inference latency to sequence length, and step-wise commitments lack native revision mechanisms.Speculative decoding and multi-token prediction alleviate latency but remain constrained by the causal paradigm.Source: §2.1 · p. 3

2.2 Discrete Diffusion Language Models

  • Discrete diffusion enables full-sequence planning and dynamic global revision, but discrete state transitions restrict continuous trajectory optimization and smooth gradient-based steering.Recent approaches scale discrete diffusion through pretraining or adapting pretrained foundations, often with blockwise decoding, re-masking, or hybrid formulations.Source: §2.2 · p. 3

2.3 Continuous Diffusion Language Models

  • Continuous diffusion generates text by iterative denoising in embedding space, but geometric misalignment with vocabulary identities causes non-uniform identity decay.Prior schedule adjustments use empirical heuristics and lack an analytical account of how representation geometry governs decay; the paper addresses this with Competition Threshold Theory and Semantic Retention Scheduling.Source: §2.3 · p. 3

3 Theoretical Analysis

The analysis formalizes token identity decay as directional pairwise competition and shows how embedding geometry and effective noise govern rank and retention. It explains why standard trajectories compress identity decay near the noisy endpoint, validates the predictions on pretrained embeddings, and motivates scheduling time by semantic retention.

3.1 Competition Threshold Theory

  • Cosine ranking removes spurious overtaking caused by embedding norm decay, and a candidate overtakes the source only when effective noise exceeds its semantic-separation threshold.The resulting Competition Threshold Theory makes source-token rank non-decreasing with noise and expresses expected rank decay through pairwise overtaking probabilities.Source: §3.1 · pp. 3–4
  • High-dimensional overtaking requires extreme noise, so standard trajectories compress identity decay near the noisy endpoint; the union-bound preservation guarantee is conservative.The theoretical scaling depends logarithmically on vocabulary size, and the bound ends before the empirically observed transition.Source: Figure 1 · p. 4, §3.1 · p. 5

3.2 Semantic Retention Scheduling

  • Logarithmic retention Qlog tracks intermediate semantic-rank decay, whereas top-1 retention collapses at the first overtake and linear retention is diluted by the vocabulary tail.At half retention, Qlog requires about √2Mend overtakes, versus Mend/2 for linear retention when the terminal overtaking count is large.Source: §3.2 · pp. 5–6
  • Factoring retention into spatial decay and the time-to-noise mapping reveals why linear trajectories allocate over 90% of optimization budget to stagnant phases.Their convex κ(γ) mapping compresses the rank-decay transition into the final O(ln N/d) trajectory fraction; semantic scheduling aligns time allocation with identity decay.Source: §3.2 · pp. 5–7

3.3 Empirical Validation

  • On pretrained embeddings, spectrum-aware predictions track measured retention statistics with half-retention errors below 2%, while the δ = 0.1 bound certifies κ ≤ 9.27 at only 1.24% decay.Measured identity decay reaches 10% at κ = 10.90; Gaussian and uniform spherical endpoint retention curves nearly coincide against effective noise magnitude.Source: Figure 2 · p. 7, §3.3 · p. 8

4 LLaCO

LLaCO adapts a pretrained backbone to continuous directional states, uses spherical geodesic noising, and reallocates time according to logarithmic rank retention. It generates text through sequential blockwise denoising conditioned on preceding clean context.

4.1 Model Architecture

  • The embedding directional adapter and timestep-conditioned modulation adapt the pretrained backbone to continuous directional states and produce token probabilities for training and clean-direction estimation.The adapter maps unit states onto the backbone’s semantic manifold through a soft convex combination of original embeddings; the model is trained with frozen token embeddings and cross-entropy.Source: §4.1 · pp. 8–9, Figure 3 · p. 9

4.2 Spherical Noising

  • Spherical noising moves unit representations along a geodesic toward a mask-biased endpoint, avoiding norm-decay artifacts while preserving directional token competition.The effective noise magnitude increases along the path and diverges near its endpoint, ensuring pathwise rank monotonicity.Source: §4.2 · pp. 9–10

4.3 Trajectory Reallocation

  • Semantic Retention Scheduling uses fitted logarithmic rank retention to allocate more training and sampling budget where rank decay is rapid.Uniform sampling in semantic time induces a density floor, preserving non-zero coverage in early and nearly flat retention phases.Source: §4.3 · pp. 10–11

4.4 Blockwise Denoising

  • Generation denoises each target block from the mask-biased endpoint along a uniform semantic-time grid, then decodes its tokens and appends them as clean context.Each update follows a spherical arc toward the model’s predicted clean direction; the update tracks the reverse noising path when that estimate matches the clean direction.Source: §4.4 · p. 11

5 Experiments

Across comparable 7B–8B models, LLaCO outperforms the evaluated continuous diffusion baselines and is competitive with discrete diffusion models. Ablations and sampling analyses examine schedule and design choices, as well as inference block size and denoising steps.

5.1 Experimental Setup

  • The evaluation covers knowledge and reasoning, mathematical problem solving, and code generation benchmarks.Models are compared at comparable 7B–8B parameter scales; training uses 1.32M open-source multi-domain instruction examples.Source: §5.1 · p. 11, §5.2 · p. 12

5.3 Comparison with Diffusion-based LLMs

  • LLaCO outperforms every evaluated continuous diffusion baseline, with absolute gains of 47.02 points on MATH and 27.44 points on HumanEval over the strongest continuous baseline.Against discrete diffusion models, it is competitive across knowledge, reasoning, and code; Dream-7B leads HumanEval by 6.10 points, while LLaCO leads MMLU and GPQA marginally and MATH by 10.92 points.Source: §5.3 · p. 12

5.4 Ablation Study

  • Replacing logarithmic rank retention with top-1 retention compresses later rank decay into a terminal window and degrades mathematical reasoning and code generation.Linear rank retention is diluted by the vocabulary tail and achieves suboptimal overall performance compared with logarithmic rank retention.Source: §5.4 · p. 12
  • Removing the mask-biased endpoint impedes transfer of the pretrained model’s inductive bias and reduces code-generation performance.Replacing spherical noising with Euclidean linear noising distorts competition and degrades performance across all evaluated tasks.Source: §5.4 · p. 12

5.5 Sampling Analysis

  • Larger inference blocks monotonically degrade generation quality across all evaluated tasks.Tokens in a block are committed simultaneously, so larger blocks require more interdependent tokens to be resolved jointly.Source: §5.5 · p. 13
  • Increasing denoising steps from 4 to 8 improves all benchmarks, while increasing them to 16 produces task-dependent fluctuations.Additional steps refine clean-direction estimates but can also divert the state from the nominal geodesic path.Source: §5.5 · p. 13

6 Conclusion

The paper develops a theory of token competition and a scheduling approach for continuous diffusion, then introduces LLaCO. Its 8B model outperforms evaluated continuous diffusion baselines and is competitive with comparable-scale discrete diffusion models.

6 Conclusion

  • LLaCO combines spherical geodesic trajectories, an embedding directional adapter, timestep-conditioned modulation, and mask-biased endpoints.Its schedule is based on logarithmic rank retention, motivated by the Competition Threshold Theory’s account of token competition and temporal compression.Source: §6 · p. 13
  • Extensive evaluations show that LLaCO outperforms all evaluated continuous diffusion baselines and is competitive with discrete diffusion models of comparable scale.The paper presents its theoretical insights and architectural designs as a principled foundation for future continuous diffusion language modeling.Source: §6 · p. 13

Limitations

The appendix supports the paper’s directional account of token competition and identifies limits on the scope of its theoretical approximations.

Limitations

  • The theoretical conclusions rely on near-orthogonal structures and high-dimensional, sparsely populated representation spaces; other geometries and noise distributions need further validation.These are the stated boundaries on applying the theory beyond the analyzed setting.Source: Limitations · p. 13

A Details of Theoretical Analysis

  • The appendix organizes supporting analyses for Section 3, including directional identity, competition thresholds, pairwise overtaking, retention statistics, and preservation bounds.It also identifies spectrum-aware and near-orthogonal predictions as support for Section 3.3.Source: A Details of Theoretical Analysis · p. 18

A.1 Spurious Overtaking under Embedding Norm Decay

  • With zero noise, a lower-norm competitor can overtake the source under Euclidean ranking as the signal shrinks, while cosine ranking preserves the source at rank 1.The Euclidean effect arises from norm differences without directional corruption.Source: §A.1 · p. 18

A.2 Competition Thresholds

  • Under the appendix’s ranking conventions, each candidate overtakes only above its threshold, making source rank non-decreasing with effective noise magnitude.The endpoint convention counts candidates whose noise score exceeds the source’s; ties do not count as overtaking.Source: §A.2 · pp. 18–19

A.3 Pairwise Overtaking under Gaussian Noise

  • For Gaussian noise, pairwise overtaking probability increases with effective noise and semantic similarity, and pairwise overtaking events determine expected rank.The analysis also couples spherical and Gaussian noise; as dimension grows, their retention and rank predictions converge.Source: §A.3 · pp. 19–20

A.4 Retention Statistics and Their Ordering

This appendix establishes how rank-based retention measures relate to overtaking counts, then develops Gaussian-noise predictions and a near-orthogonal surrogate.

A.4 Retention Statistics and Their Ordering

  • The exact retention statistics are ordered as top-1 retention ≤ logarithmic retention ≤ linear retention along trajectories with non-decreasing noise and at least one terminal overtaking.The logarithmic score emphasizes leading ranks because crossing rank level q has weight approximately 1/q.Source: §A.4 · pp. 20–21
  • For large terminal overtaking counts, logarithmic retention reaches a given level after a vanishing fraction of terminal candidates has overtaken, while continuing to decline until all crossings occur.Unlike the top-1 indicator, it therefore captures later rank deterioration.Source: §A.4 · p. 21

A.5 Retention Spectra under Gaussian Noise

  • Conditioning on the shared source projection separates the common noise component that couples candidate overtaking events, enabling conditional count predictions.For nearly orthogonal directions, the correlation between relative noise advantages approaches 1/2 before this conditioning.Source: §A.5 · pp. 21–22
  • The independent-residual Gaussian approximation controls count-mean error, but does not bound additional error from residual correlations; its plug-in lower comparison with exact top-1 retention need not hold.The exact ordering of the statistics themselves remains valid.Source: §A.5 · p. 24

A.6 Near-Orthogonal Surrogate

  • In the near-orthogonal surrogate, the three statistics depend only on dimension and vocabulary size at a given noise level, and decrease from 1 to 0 as noise grows without bound.The model sets pairwise similarities to zero and yields an exact linear-retention expression within that surrogate.Source: §A.6 · p. 25

A.7 First-Overtaking Scale

The appendix characterizes initial overtaking with quantiles and maps its noise-scale transition into trajectory time, showing how trajectory geometry shapes temporal compression.

A.7 First-Overtaking Scale

  • The first-overtaking transition is characterized by quantiles because its threshold has an infinite mean, even conditional on being finite.The quantile-based width is mainly set by the source projection rather than fluctuations of the candidate maximum.Source: §A.7 · p. 26
  • For d = 4096 and N = 130,067, numerical integration gives κ0.5 ≈14.4 and Wκ ≈9.4, so initial identity decay requires effective noise about an order of magnitude above 1.The probit approximation gives κ0.5 ≈14.4 and Wκ ≈9.1; its stated bound gives κ0.5 ∈[14.3, 14.5].Source: §A.7 · p. 26

A.8 Trajectory Factorization and Time Windows

  • Reparameterizing time preserves which competitions occur and their order; compression instead arises because effective noise rises rapidly where it is large.For linear trajectories, the fraction of time above effective noise κ is 1/(1 + κ).Source: §A.8 · p. 27
  • For d = 4096 and N = 130,067, the linear trajectory gives γ0.5 ≈0.935 and Wγ ≈0.036, placing the transition near the noisy endpoint.Its transition occupies a window whose width scales as O(d^-1/2).Source: §A.8 · p. 27
  • Spherical interpolation gives γ0.5 ≈0.956 and Wγ ≈0.026 for the same d and N, compressing the transition by about 2/π relative to the linear trajectory.For nearly orthogonal endpoints, κ(γ) ≈ tan(πγ/2).Source: §A.8 · p. 28

A.9 Preservation Bounds and Shared Components

This appendix develops sufficient preservation guarantees and shared-component scaling, then evaluates token geometry and competition predictions on pretrained embeddings. The experiments find a near-orthogonal bulk alongside sparse higher-similarity neighbors.

A.9 Preservation Bounds and Shared Components

  • At amax = 0 and δ = 0.1, the near-orthogonal surrogate gives κ0.9 ≈ 11.1, compared with a bound of about 9.42; these indicate the guaranteed preservation range and δ-decay location, respectively.The right-hand value is where the decay probability reaches δ.Source: §A.9 · p. 29
  • With equal-coefficient shared components, threshold quantiles and widths scale by √(1 − ρ), leaving their relative width unchanged.Exact scaling requires equal coefficients and consistent source, noise, and amplitude conventions; token-dependent coefficients generally break it.Source: §A.9 · pp. 28–29

B Empirical Analysis of Token Geometry and Competition

B.1 Token Geometry Across Pretrained Models

  • Across measured models, mean pairwise cosine ranges from 0.0161 to 0.0300, while mean maximum cosine ranges from 0.2316 to 0.4940.Between 94.838% and 99.912% of evaluated pairs have absolute cosine at most 0.1, yet each source has about 0.37–12.27 candidates above cosine 0.3.Source: §B.1 · p. 30

B.2 Paired Competition Experiment: Setup and Uncertainty

  • The paired competition experiment samples 4,096 Nemotron sources with 128 independent noise vectors each and compares measured statistics with spectrum-aware and near-orthogonal predictions.The predictions are not fitted to the empirical curve; uncertainty intervals use 1,000 source-clustered bootstrap replicates.Source: §B.2 · pp. 30–31

B.3 Competition Statistics in Effective-Noise Coordinates

In effective-noise coordinates, retention statistics show successive stages of identity decay, and theoretical predictions track the measured curves. The appendix also compares trajectory time profiles, preservation bounds, and mask-biased endpoint behavior.

B.3 Competition Statistics in Effective-Noise Coordinates

  • Gaussian half-retention occurs at κ = 14.169 for P, 23.670 for Qlog, and 89.688 for Q.Qlog responds beyond first identity decay and reaches its midpoint well before linear retention Q.Source: Table 4 · p. 31, §B.3 · p. 32

B.4 Trajectory Parameterization and Temporal Compression

  • For Qlog, the measured 90%–10% transition width is 0.0542 on the linear trajectory and 0.0370 on the spherical trajectory.Spherical degradation occurs later and over a narrower interval in this experiment, although paired endpoints produce similar curves in effective-noise coordinates.Source: §B.4 · p. 32

B.5 Preservation Bounds and Observed Transitions

  • At δ = 0.1, the full-spectrum union bound gives κ = 9.26550, versus 10.89777 for the measured first-overtaking quantile.Measured decay at the bound is 1.24035%, below the allowed 10%, so the union bound remains conservative with actual candidate similarities.Source: §B.5 · p. 32, Table 5 · p. 32

C.1 Trajectory and Endpoint Properties

  • Under the mask-biased endpoint, candidates more aligned with the mask direction have stochastically smaller competition thresholds when semantic separation is equal.The approximation also predicts at least a twofold reduction in relative-noise variance for candidates with nonpositive mask-direction advantage; the schedule itself uses Qlog measured under the exact endpoint.Source: §C.1 · pp. 33–34

C.2 Retention Curve Fitting

The appendix fits retention curves to construct the semantic-time mapping, then describes the training and sampling implementation. The reverse update follows the geodesic toward the estimated clean direction.

C.2 Retention Curve Fitting

  • The endpoint-normalized Logistic has the smallest reported error among the tested fitting families for Qlog.Its fitted normalized curve has a half-retention location of approximately 0.963 and a 90%–10% transition width of approximately 0.059.Source: §C.2 · p. 35, Table 6 · p. 35

C.3 Implementation Details

  • Training uses an 8-token block layout, with noisy positions conditioned on preceding clean blocks and trained without sequentially unrolling the blocks.Noisy positions receive their block’s semantic time; the shared inverse mapping converts it to geometric position.Source: §C.3 · p. 36

C.4 Sampling Procedure

  • At inference, each block starts at the mask-biased endpoint and is denoised on a uniform semantic-time grid mapped to geometric positions.Intermediate steps estimate the clean direction from the 512 highest-probability tokens and move along the geodesic; the final evaluation takes the argmax token.Source: §C.4 · p. 37
  • If the current state lies on the true trajectory and the clean-direction estimate is exact, the reverse update reaches the true next trajectory state.This exactness is conditional on those assumptions.Source: §C.4 · p. 37

D.1 Training and Sampling Settings

The appendix details LLaCO’s training and sampling settings and explains which model families are included in the evaluation. Comparisons focus on same-scale diffusion models under the supported evaluation protocol.

D.1 Training and Sampling Settings

  • Training runs for 165k optimization steps on 1.32M examples, processing approximately 1.685B text tokens, including context.Approximately 1.300B target tokens, including EOS padding, are supervised; these figures exclude backbone pretraining.Source: §D.1 · p. 37
  • Training mixes dialogue, mathematics, science, and code at 30%/30%/30%/10% by example count, using 8-token blocks.Default inference uses 4-token blocks and four model evaluations per block, with a semantic-time grid from 1 to 0.01.Source: §D.1 · p. 38

D.2 Model Landscape and Baseline Selection

  • LLaCO is compared with same-scale discrete and continuous diffusion models; autoregressive and hybrid models are listed but excluded.The continuous baselines TESS-2 and DSL-LLaDA are evaluated under the same protocol, while other models are excluded for scale, tuning, or evaluation-setting mismatches.Source: §D.2 · pp. 38–39

D.3 Block Size and Sampling Steps

In LLaCO-8B, inference block size has a much stronger effect on generation quality than steps per block. Smaller blocks perform better, while increasing steps from four to eight improves all four tasks before further increases produce task-dependent changes.

D.3 Block Size and Sampling Steps

  • The comparison uses the default LLaCO-8B checkpoint, holds other decoding settings fixed, and uses (4, 4) as the default configuration.The study varies N = B ∈{4, 8, 16}, then fixes B = 8 and varies N ∈{4, 8, 16}.Source: §D.3 · p. 39
  • GSM8K falls from 80.74 to 48.07 and MATH from 62.12 to 24.28 as block size grows from (4, 4) to (16, 16).HumanEval and MBPP also lose about 33 and 26 points, respectively; the (4, 4) setting outperforms (8, 8), despite matching training's 8-token blocks.Source: §D.3 · p. 39
  • At B = 8, increasing N from 4 to 8 improves all four tasks, while N = 16 yields slight declines on all but HumanEval.GSM8K and MATH vary by less than 3.7 points across N ∈{4, 8, 16}, much less than the variation induced by block size.Source: §D.3 · p. 39
  • The Qlog schedule concentrates evaluations near γ ≈0.96, so with N = 4, intermediate evaluations already cover the main identity-decision window.More steps mostly subdivide that interval; although a geodesic update is exact with the true clean direction, estimated directions can introduce step-size errors, and the authors hypothesize a trade-off between refinement and trajectory deviation.Source: §D.3 · p. 39
  • Larger blocks place more mutually dependent positions in a joint decision, with dependencies resolved through continuous-state evolution rather than conditioning on accepted tokens.This is consistent with the strong block-size effect and the reported block-size trend for discrete block diffusion.Source: §D.3 · p. 39

Related papers

Loading 2610.32604v1…