PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun

arXiv:2605.18530v2cs.CLcs.AIcs.LGstat.ML

TL;DR

Prior work reported a scalability gap between continuous and discrete diffusion language models. This paper revises Plaid as RePlaid and compares it under an aligned scaling protocol. RePlaid scales competitively with discrete models, while likelihood-based training yields a near-linear noise schedule and structured embedding geometry.

  • Problem

    Prior evaluations found continuous diffusion underperformed discrete diffusion and autoregressive models at matched training FLOPs and parameter counts, leaving its scalability uncertain.

    Source: §1 · p. 2

  • Method

    The paper revises Plaid as RePlaid, aligns its transformer architecture with modern discrete DLMs, and evaluates it using a unified scaling protocol.

    Source: §1 · p. 2

  • Results

    RePlaid scales competitively with discrete DLMs, with a 20× compute gap versus AR at matching validation loss when self-conditioning is used.

    Source: §3.4 · p. 5

  • Takeaways & Limitations

    Likelihood-based continuous DLMs can rival discrete models in likelihood and sample quality when architecture and experimental setting are properly aligned.

    Source: §6 · p. 11

    A 20× compute gap from AR remains, and whether the scaling trends persist at multi-billion-parameter scale remains to be investigated.

    Source: §6 · p. 11

Abstract

from arXiv · show

While diffusion has drawn considerable recent attention from the language modeling community, continuous diffusion has appeared less scalable than discrete approaches. To challenge this belief we revisit Plaid, a likelihood-based continuous diffusion language model (DLM), and construct RePlaid by aligning the architecture of Plaid with modern discrete DLMs. In this unified setting, we establish the first scaling law for continuous DLMs that rivals discrete DLMs: RePlaid exhibits a compute gap of only $20\times$ compared to autoregressive models, outperforms Duo while using fewer parameters, and outperforms MDLM in the over-trained regime. We benchmark RePlaid against recent continuous DLMs: on OpenWebText, RePlaid achieves a new state-of-the-art PPL bound of $22.1$ among continuous DLMs and superior generation quality. These results suggest that continuous diffusion, when trained via likelihood, is a highly competitive and scalable alternative to discrete DLMs. Moreover, we offer theoretical insights to understand the advantage of likelihood-based training. We show that optimizing the noise schedule to minimize the ELBO's variance naturally yields linear cross-entropy (information loss) over time. This evenly distributes denoising difficulty without any case-specific time reparameterization. In addition, we find that optimizing embeddings via likelihood creates structured geometries and drives the most significant likelihood gain.

1 Introduction

Continuous diffusion offers potential advantages for controllability and efficient generation, but prior scaling results raised doubts about its competitiveness. The paper revisits Plaid with an aligned architecture and reports competitive scaling, generation quality, and likelihood-based training insights.

1 Introduction

  • Continuous diffusion's smooth latent geometry supports structured editing and ODE-based solvers for efficient generation.The paper also notes that continuous diffusion can trade computational cost against sample quality under limited sampling budgets.Source: §1 · p. 1
  • Earlier evaluations found continuous diffusion underperformed discrete diffusion and AR at matched training FLOPs and parameter counts, with Plaid requiring 64× AR compute for matching likelihood [3] [18].The paper asks whether that gap reflects an inherent limitation or incompatible scaling analyses.Source: §1 · p. 2
  • 20× compute separates RePlaid with self-conditioning from AR at matching validation loss under a unified scaling protocol.The protocol follows Sahoo et al. [51] and aligns the dataset, optimization hyperparameters, and compute budget.Source: §1 · p. 2
  • RePlaid revises Plaid with a transformer architecture aligned to modern discrete DLM practices.This alignment supports a unified comparison using the protocol from Sahoo et al. [51].Source: §1 · p. 2
  • ELBO-based training naturally recovers a near-linear cross-entropy noise schedule and regularizes embedding geometry for likelihood improvement.The paper says the schedule avoids heuristic time reparameterization and embedding regularization prevents potential token dispersion.Source: §1 · p. 2
  • 22.1 perplexity on OpenWebText sets RePlaid's state-of-the-art bound among continuous DLMs.The paper also reports that RePlaid rivals discrete DLMs in likelihood and sample quality.Source: §1 · p. 2

2 Background: Plaid—A Likelihood-Based Continuous DLM

Plaid is a likelihood-based variational diffusion model for text that embeds tokens in a learned continuous space and applies Gaussian noising. Its denoiser and training objective combine reconstruction, diffusion, and prior terms, with a learnable noise schedule and self-conditioning.

2 Background: Plaid—A Likelihood-Based Continuous DLM

  • Plaid represents each token with a learned unit-length embedding in a low-dimensional space, using dimension 16 unless otherwise stated.A sequence of one-hot token vectors x is embedded as e = xE.Source: §2 · p. 2
  • The forward process applies Gaussian noising to the embedded sequence, with smooth time-dependent scalar functions α_t and σ_t.Plaid is introduced as a Variational Diffusion Model for text.Source: §2 · p. 2
  • The time-conditioned denoiser predicts a categorical distribution over vocabulary tokens at every sequence position.Before softmax, output prior logits from a closed-form Gaussian log-density help focus predictions on plausible tokens given z_t.Source: §2 · pp. 2–3
  • The negative evidence lower bound comprises prior, reconstruction, and diffusion losses.Training allocates part of each batch to reconstruction-loss estimation and the remainder to diffusion-loss estimation; the prior loss uses the whole batch.Source: §2 · p. 3
  • Plaid learns the noise-schedule endpoints and trains its interior shape to minimize diffusion-loss Monte Carlo variance.The schedule's interior shape is a monotonically increasing neural network.Source: §2 · p. 3
  • Self-conditioning feeds an initial clean-data estimate back into the denoiser for 25% of training examples and all validation and sampling examples.The initial estimate comes from a gradient-free forward pass.Source: §2 · p. 3
  • For the stated GPT-2-scale example, Plaid's embedding operations require approximately 50× fewer FLOPs than projecting one-hot embeddings to hidden size.The comparison uses vocabulary size about 50K, hidden size 768, and embedding dimension 16.Source: §2 · p. 3

3 Revisiting Scaling Laws of Continuous DLMs

The section revises Plaid’s transformer to align with modern discrete DLMs and compares it with discrete models under a unified scaling protocol. RePlaid scales competitively and shows parameter-efficiency and over-training advantages in the reported settings.

3.1 RePlaid: Aligning Plaid with Modern Discrete DLM Architecture

  • RePlaid retains Plaid’s algorithmic components while matching MDLM and Duo’s transformer design, including LayerNorm, GELU(tanh), and AdaLN-Zero modulation.The authors also remove Plaid’s FP32 final-logit-head difference to avoid a numerical confounder.Source: §3.1 · p. 3

3.2 A Unified Scaling Benchmark for Continuous and Discrete DLMs

  • The benchmark uses deduplicated SlimPajama, a shared AdamW configuration, and offsets self-conditioning’s ≈8.3% non-embedding FLOP increase by reducing training steps.Models use batch size 256 and sequence length 2048; RePlaid’s optimizer adjustments accommodate its learnable schedule and embeddings.Source: §3.2 · p. 4

3.3 IsoFLOP Analysis and Scaling Laws

  • IsoFLOP analysis fits validation loss across model sizes at fixed compute budgets, then fits a power law to compute-optimal loss; RePlaid’s curves are well behaved.The fixed-budget loss curves use quadratic fits in log-log space, and diffusion losses use variational upper bounds on negative log-likelihood.Source: §3.3 · p. 4

3.4 Results

  • ≈20× compute gap to match AR for RePlaid (s.c.) approaches MDLM’s 14× and improves on Duo’s 22×, with comparable power-law loss scaling.RePlaid without self-conditioning has a 27× compute gap.Source: Figure 2 · p. 5, §3.4 · p. 5
  • ≈1.8× fewer parameters than MDLM and Duo lets RePlaid (s.c.) outperform Duo’s compute-optimal loss; RePlaid also uses 3.4× fewer parameters than AR.These comparisons concern RePlaid’s compute-optimal frontier.Source: Figure 2 · p. 5, §3.4 · p. 5
  • At a fixed model size, RePlaid (s.c.) trained to ≈3.1× its optimal compute matches MDLM trained to ≈6.9×, and has lower loss in the over-trained regime.The over-trained regime is below the green line in Figure 1.Source: Figure 1 · p. 4, §3.4 · p. 5
  • Applying RePlaid’s low-dimensional, length-normalized embeddings and related adjustments to MDLM degrades performance, and the reverse transfer does too.The authors connect this asymmetry to low-rank embedding structures learned by RePlaid but not MDLM.Source: §3.4 · p. 5

4 Benchmarking RePlaid for Diffusion Language Models

At the canonical DLM scale, the paper compares RePlaid with discrete and continuous models on likelihood and sample quality. RePlaid achieves the best reported OWT PPL bound among considered DLMs, while its sampling results vary with sampler and step budget.

4.1 Likelihood Evaluation

  • 22.1 PPL bound on OWT gives RePlaid (s.c.) the best result among considered DLMs, ahead of MDLM (low var., 23.1) and Duo (25.2).On LM1B, RePlaid (s.c.) reaches a top-2 PPL of 31.6, surpassing Duo but not MDLM.Source: Table 1 · p. 6, §4.1 · p. 6

4 Benchmarking RePlaid for Diffusion Language Models

  • The benchmark compares likelihood and sample quality against leading discrete and continuous DLMs, using a shared transformer architecture and comparable 1M-step optimization settings.The authors treat both evaluation axes as necessary because better likelihood may not always translate into better sample quality.Source: §4 · p. 6

4.1 Likelihood Evaluation

  • Freezing token embeddings to random values degrades PPL from 22.1 to 39.4; the noise schedule, self-conditioning, and output prior contribute gains of ∆= 2.3, ∆= 1.5, and ∆= 0.4, respectively.Omitting any one of the noise schedule, self-conditioning, or output prior still leaves RePlaid ahead of Duo.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 Sampling

  • As sampling steps increase, continuous methods generally match or outperform Duo on GenPPL; RePlaid (s.c.) leads from T = 64 and reaches ≈21 at T = 1024.On MAUVE, RePlaid (no s.c.) outperforms Duo at every T; RePlaid (s.c.) has the lowest entropy at high T, which the authors attribute to self-conditioning sharpening its distribution.Source: Figure 3 · p. 7, §4.2 · p. 8
  • At small T, DPM-Solver++(2M) beats DDPM and DDIM, but from T ≥64 DDPM overtakes deterministic solvers and reaches the highest MAUVE.The authors report that Heun is less favorable than DPM-Solver++(2M), likely because each Heun step costs 2 NFEs rather than 1.Source: §4.2 · p. 8, Figure 4 · p. 8

5 Theoretical Insights

The section explains how likelihood-based ELBO training shapes embedding geometry and how learning the noise schedule distributes denoising difficulty over time. It contrasts these effects with auxiliary cross-entropy training and derives conditions for near-linear per-timestep cross-entropy.

5.1 ELBO versus Cross-Entropy Training

  • 22.1 versus 26.1 PPL bound: adding CE flattens the embedding PCA spectrum and degrades RePlaid’s bound, while ELBO training yields POS-structured, low-rank embeddings.At embedding dimension 16, the embeddings concentrate on roughly six principal components, accounting for 90% of cumulative explained variance.Source: §5.1 · p. 9, Figure 5 · p. 9
  • RePlaid’s likelihood objective is an ELBO; adding CE applies a separative force to embeddings that can disrupt their low-rank structure and hurt the PPL bound.The comparison uses a combined NELBO and CE loss, with the CE gradient constrained to reach the embeddings and denoiser.Source: §5.1 · p. 9, §5 · p. 9
  • At embedding dimension 768, RePlaid has lower-rank embeddings than LangFlow alongside a better PPL bound, while MDLM learns higher-rank embeddings than both.These comparisons are reported in Figures 14 and 15.Source: §5.1 · p. 9

5.2 Benefits of Learning the Noise Schedule

  • With a learned noise schedule, per-timestep diffusion loss is near-constant and cross-entropy is near-linear; this pattern was validated across varied embeddings.The empirical analysis also reports near-linear cross-entropy in the middle of training, despite the theoretical residual term.Source: Figure 6 · p. 10, §5.2 · p. 11
  • A unique noise schedule under fixed endpoints makes per-timestep diffusion loss constant and its variance zero.The proposition defines the constant loss as the expected timestep loss, κ.Source: §5.2 · p. 10
  • Under a Bayes-optimal denoiser and no embedding collapse, per-timestep cross-entropy decomposes into a linear trend and a non-negative, increasing conditional-total-correlation residual.The linear trend follows from mutual information decaying linearly with time at a rate set by the diffusion loss.Source: §5.2 · pp. 9, 11

6 Conclusion and Discussion

The paper concludes that continuous diffusion scales on par with discrete models when architecture and experimental settings are aligned, and presents RePlaid as a likelihood-based model with competitive results. It also summarizes related discrete and continuous diffusion approaches and identifies remaining scaling questions.

6 Conclusion and Discussion

  • RePlaid’s ELBO-based approach achieves a 22.1 PPL bound on OpenWebText and rivals discrete DLMs in likelihood and sample quality; the paper reports continuous models scale on par when properly aligned.The ELBO is described as naturally recovering near-linear cross-entropy over time and regularizing embedding geometry.Source: §6 · p. 11
  • A 20× compute gap to autoregressive models remains, and whether the scaling trends persist at multi-billion-parameter scale is an identified next question.The discussion also names learnable token embeddings as the primary driver of RePlaid’s perplexity gains.Source: §6 · p. 11

A Related Works

  • The related-work survey distinguishes discrete diffusion on tokens from continuous diffusion on continuous token representations, while using autoregressive models as the scaling-law reference baseline.It explicitly does not attempt to review the dominant autoregressive paradigm.Source: A Related Works · p. 19

A.1 Discrete diffusion language models

  • Discrete DLM work spans masked and score-matching objectives, hybrids with autoregressive decoding, discrete flow methods, and models scaled to 7–8B parameters; the paper adopts the IsoFLOP protocol of Nie et al. [38] and Sahoo et al. [51].The survey identifies MDLM [48], MD4 [55], and RADD [41] as independently deriving a clean per-position cross-entropy objective from the absorbing-state model.Source: §A.1 · p. 19

A.2 Continuous diffusion language models

  • Continuous approaches include embedding-space diffusion, simplex or statistical-manifold flows, and dequantization-based methods; RePlaid revisits Plaid [18] under the IsoFLOP protocol to examine the source of its reported compute gap.The survey also describes FLM [29] and DFM [44] as using fixed one-hot embeddings and cross-entropy, while LangFlow [10] uses a pure cross-entropy objective with learnable embeddings.Source: §A.2 · p. 19

B Training Algorithm

This section documents Plaid’s training procedure and sequence-level NELBO, along with sampling and ODE-based likelihood estimation and its limitations under self-conditioning. It also examines how omitting a chain-rule trace in cached Hutchinson estimation affects likelihood and reported perplexity.

B Training Algorithm

  • Plaid splits each minibatch between reconstruction at t = 0 and diffusion at low-discrepancy times, while the prior uses every example.The reconstruction share adapts to the running ratio of reconstruction and diffusion loss standard deviations. Self-conditioning is enabled at psc = 0.25; schedule endpoints minimize diffusion loss, while the interior schedule shape is updated to minimize its Monte-Carlo variance using gradients from diffusion rows only.Source: B Training Algorithm · p. 20

C Derivation of the sequence-level NELBO for Plaid

  • Jensen’s inequality yields a sequence-level NELBO decomposed into prior, reconstruction, and diffusion losses.The per-step KL terms reduce to scaled squared distances between predicted and ground-truth embeddings; in continuous time, the SNR derivative makes the diffusion loss non-negative.Source: C Derivation of the sequence-level… · pp. 21–22

D Sampler Update Formulas

  • The sampler appendix covers one stochastic ancestral sampler and three deterministic PFODE solvers using a variance-preserving schedule, with final tokens read out from decoder logits.Each step predicts clean data; the trajectory runs from t = 1 to 0, followed by an argmax at t ≈ 0.Source: D Sampler Update Formulas · p. 23

D.1 Stochastic Ancestral Sampler (DDPM)

  • The default DDPM ancestral sampler draws each reverse state from the closed-form Gaussian posterior and uses T + 1 network evaluations including the final argmax.Its per-step SNR gap is c = 1 − SNR(t)/SNR(s).Source: §D.1 · p. 23

D.2 Deterministic PFODE Solvers

  • The deterministic PFODE solvers differ by discretization: DDIM is exact at any step size when its clean-prediction linearization is exact, DPM-Solver++(2M) extrapolates cached predictions, and Heun uses predictor–corrector steps.DDIM and DPM-Solver++(2M) use T + 1 evaluations including argmax; Heun uses 2T.Source: §D.2 · pp. 23–24

E ODE-based Likelihood Estimation

  • The ODE-based likelihood estimator is an internal sanity check on VDM NELBO values, not a source of headline results.Its importance-weighted estimate converges almost surely as K → ∞, is non-decreasing in K, and has bias Θ(1/K).Source: E ODE-based Likelihood Estimation · p. 25, §E.1 · p. 25

E.1 ODE-based likelihood identity

  • The IWAE estimate and VDM NELBO cannot be directly ordered because they target different model densities; their empirical crossover reflects mismatch between SDE- and PFODE-pushforward marginals.Within the PFODE bound, the IWAE family tightens monotonically in K. The reported PPL uses a γ-space log-weight, which avoids schedule-induced stiffness near t → 0, 1.Source: §E.1 · pp. 26–27

E.2 Self-conditioning and the Chain-rule Trace

  • With self-conditioning, caching the bootstrap prediction during trace estimation drops a chain-rule term, so the resulting estimate is not a Liouville identity or an upper bound on −log pθ(x).Recompute self-conditioning at perturbed states or retain the bootstrap forward pass in the JVP graph, at one extra backbone forward per perturbation; without self-conditioning, the standard estimator is unbiased.Source: §E.2 · p. 27
  • When self-conditioning pathways agree, the omitted chain-rule trace is typically negative, so dropping it overestimates integrated divergence, inflates log pθ(z0) and log-weight, and deflates reported PPL.The bias grows with ∂zxsc and is largest in the high-SNR regime, where the bootstrap is most accurate.Source: §E.2 · p. 27

E.3 Experimental Results and Ablation Study

Estimator checks show that omitting the chain-rule correction understates likelihood estimates, while increasing the IWAE sample count tightens them. The theory identifies a unique constant-loss noise schedule and, under optimality, linear information decay.

E.3 Experimental Results and Ablation Study

  • 8.03 PPL on RePlaid and 8.35 on LangFlow are deflated when the chain-rule term is dropped; only the corrected estimate is provably an upper bound.The comparison covers PPL estimation methods on OpenWebText.Source: Table 3 · p. 28, §E.3 · p. 28
  • 22.08 for RePlaid and 29.11 for LangFlow are fitted PPL∞ asymptotes; RePlaid meets matched VDM NELBO at K=32 (22.11 vs. 22.1).The fitted a + b/K curves match the data for K ≥ 4; paper-quality likelihood reporting should use K ≳ 16.Source: §E.3 · p. 28, Figure 7 · p. 29
  • Mean PPL remains within 0.4 of baseline 23.54 across the tested solver and Hutchinson variants.The results support convergence of heun2-128 and indicate that the remaining scatter comes from K=1 importance-sampling variance.Source: §E.3 · p. 28, Table 4 · p. 29

F Constant Per-Timestep Diffusion Loss

  • A unique noise schedule satisfying the endpoint constraints makes per-timestep diffusion loss constant and its time variance zero.The proposition defines the constant loss as κ = E_t[ℓ_θ,γ(t)].Source: F Constant Per-Timestep Diffusion Loss · p. 30

G Linear Information Decay Under Optimality

  • Under the optimal schedule and a Bayes-optimal model, information I(e; z_t) decays linearly as I(e; z_0) − κt.The result follows by integrating the constant-loss condition over time.Source: G Linear Information Decay Under… · p. 31

H Per-Timestep CE Under Optimality

Under no embedding collapse, per-timestep cross-entropy has a linear trend plus an increasing conditional-dependence residual. The schedule figure separates endpoint parameters from the interior shape.

H Per-Timestep CE Under Optimality

  • Under no embedding collapse, per-timestep CE has a linear trend plus a conditional-total-correlation residual that increases monotonically over time.The residual lies between zero and the unconditional total correlation C(x).Source: H Per-Timestep CE Under Optimality · p. 32
  • Conditional total correlation increases strictly whenever any token pair remains correlated under the Bayes posterior.It is zero exactly when tokens are conditionally independent given z_t.Source: H Per-Timestep CE Under Optimality · pp. 32–33

I Learning Noise Schedule in VDMs

  • Noise-schedule endpoints and the interior shape are parameterized separately, with the endpoints and shape trained against diffusion loss and estimator variance, respectively.Source: Figure 8 · p. 34

J Aligning Plaid with Modern Discrete DLM Architecture

The paper documents RePlaid’s architecture alignment and precision choices, alongside optimization settings used for scaling-law experiments and 0.1B-scale runs.

J Aligning Plaid with Modern Discrete DLM Architecture

  • Table 5 records the component alignment of Plaid with modern discrete diffusion architecture to form RePlaid.The table marks components used in the alternative method in gray.Source: Table 5 · p. 35
  • The learnable noise schedule uses float64 for numerical stability, monotonicity, and differentiability; dtype promotion carries it into final loss aggregation.Source: Table 6 · p. 35

K Optimization Details

  • RePlaid’s embedding, monotone-MLP, and noise-endpoint learning rates and weight decay follow Plaid; lower learning rates hurt RePlaid at short training horizons.The optimization tables also give configurations for scaling-law experiments and 0.1B-scale runs.Source: Table 7 · p. 36, Table 8 · p. 36, Table 9 · p. 36

L IsoFLOP Analysis Transformer Configurations

The analysis reports IsoFLOP scaling comparisons and examines whether exchanging the embedding configurations changes the methods’ performance.

L IsoFLOP Analysis Transformer Configurations

  • Table 10 lists the AR, MDLM, and RePlaid transformer configurations used in the scaling-law study, following Nie et al. and Sahoo et al.Source: Table 10 · p. 37

M IsoFLOP Curves

  • 14× for MDLM (low var.), 22× for Duo, 20× for RePlaid (s.c.), and 27× for RePlaid (no s.c.) are worse scaling than AR; over-trained RePlaid (s.c.) beats MDLM (low var.).The IsoFLOP curves plot optimal model sizes under fixed compute budgets.Source: Figure 9 · p. 38

N Exchanging Embedding Configurations of MDLM and RePlaid

  • Exchanging embedding configurations degrades both MDLM and RePlaid (s.c.); the tested RePlaid setup uses 768-dimensional embeddings without length normalization.The exchanged MDLM setup uses 16-dimensional embeddings with a linear up-projection to 768 dimensions and length normalization.Source: Figure 10 · p. 39, N Exchanging Embedding Configurations… · p. 39

O Training Dynamics

The appendix reports that learning the noise schedule produces near-linear per-timestep cross-entropy across embedding geometries, alongside analyses of sampling, embeddings, and POS-tagging procedures.

O Training Dynamics

  • The reconstruction-loss share of the global batch is tied across 16 GPU ranks, so it changes only in 3.125% increments.For global batch size 512, each rank has a batch size of 32.Source: Figure 11 · p. 40

P Sampling with Temperature

  • At τ = 1, RePlaid is competitive with discrete DLMs and surpasses prior continuous DLMs; MAUVE supplements GenPPL to account for low-entropy collapse.Temperature sweeps trace GenPPL–entropy frontiers; high τ produces degenerate samples for both LangFlow and RePlaid with self-conditioning.Source: Figure 12 · p. 41, P Sampling with Temperature · p. 41, Figure 13 · p. 41

Q Computing Dominant POS Tags per GPT-2 Subword

  • The POS-tag analysis assigns each GPT-2 subword a dominant tag by aligning its character span to a spaCy word span.Subwords inherit the tag of the word with which their span maximally overlaps.Source: Q Computing Dominant POS Tags per… · p. 42

Q.1 Pipeline

  • The pipeline decodes token rows, tags the text, re-encodes subwords with character spans, aligns tags, and tallies counts across 18 tags.Re-encoding helps align spans when dataset chunk boundaries truncate multi-byte UTF-8 sequences.Source: §Q.1 · p. 42

Q.2 Worked Example

  • Character-span alignment assigns both subwords of “purring” the VERB tag, while whitespace-only subwords receive a SPACE tag.The example uses majority overlap with spaCy word spans.Source: §Q.2 · p. 43

Q.3 Aggregation

  • The full token-by-tag count matrix supports POS-entropy analyses in addition to assigning each observed token its most frequent tag.Tokens never observed receive no dominant tag.Source: §Q.3 · p. 43

R PCA Scree for de = 768

  • RePlaid with self-conditioning has lower-rank embedding geometry than LangFlow while achieving a better PPL bound.The comparison uses de = 768 and length-normalized embeddings.Source: Figure 14 · p. 44

S Near-Linear Per-Timestep Cross-Entropy Loss for Diverse Embeddings

  • Learning the noise schedule yields near-linear per-timestep cross-entropy regardless of embedding geometry.The appendix also reports validation PPLs for learned, randomly initialized frozen, BERT, and one-hot embeddings.Source: S Near-Linear Per-Timestep… · p. 45, Figure 16 · p. 45

T Text Samples Generated by RePlaid Trained on OWT

The section presents un-cherry-picked RePlaid samples generated after training on OpenWebText and highlights the authors’ assessment of their long-range topical consistency.

T Text Samples Generated by RePlaid Trained on OWT

  • Un-cherry-picked samples of length 1024 from RePlaid trained for 1M steps on OpenWebText are presented as demonstrating strong long-range topical consistency.Document boundaries and entities or concepts tracked across sentences and paragraphs are highlighted.Source: T Text Samples Generated by RePlaid… · p. 46
  • The sample text is flagged as containing unmoderated content, including fictionalized violence and sensitive political references.Source: T Text Samples Generated by RePlaid… · p. 46
Loading 2605.18530v2…