Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation

1 Introduction

• Diffusion models can lose generation quality when their successive denoising passes are reduced, while processing the full sequence at every step limits speedups.Bidirectional attention also prevents direct reuse of autoregressive-style KV caching.Source: §1 · p. 2
• Few-step samplers and distillation add training stages, can reduce sample diversity, and leave per-step cost unchanged; block-autoregressive diffusion permits KV caching but sacrifices parallelism across blocks.Block-autoregressive blocks must be generated sequentially, with each waiting for the previous block.Source: §1 · pp. 2–3
• NCM encodes text into K levels of discrete codes, from short coarse representations to longer fine ones, then predicts all codes within each level in parallel, without distillation.The levels are generated sequentially from coarse to fine.Source: §1 · p. 3
• Across TinyStories, LM1B, and OpenWebText, NCM matches or outperforms strong diffusion baselines with approximately 4–11× fewer function evaluations, and it outperforms AR and diffusion baselines on Sudoku across difficulty levels.On Sudoku, it also uses fewer function evaluations than any of the compared baselines.Source: §1 · p. 3
• 10–31× faster generation at comparable or better quality; using the saved time for candidate generation and MBR decoding yields about 7% higher ROUGE-L than the strongest summarization baseline within the same sampling-time budget.The speed comparison is reported in Figures 1 and 5; the MBR result is reported in Figure 6.Source: §1 · p. 3
• 4–5× fewer training tokens than strong diffusion baselines on LM1B and OpenWebText, including tokenizer and generator training, while maintaining competitive generation quality.The reported comparison is associated with Figure 7 and Appendix C.Source: §1 · p. 3
• Preliminary results report improved LM1B generation quality with larger NCM models at a fixed training-token budget, while qualitative probes suggest coarse levels capture global structure and finer levels add detail.The scaling result is reported in Figure 6a; the qualitative probes are in Appendix H, Figures 17–19.Source: §1 · p. 3

Diffusion & Flow-Based Models for Text Generation

• Unlike block-autoregressive diffusion, which generates blocks sequentially, NCM predicts a learned hierarchy of levels, with parallel prediction within each level.Block-autoregressive diffusion allows KV caching, but each block waits for the previous one.Source: §2 · pp. 3–4
• NCM generates a learned hierarchy of discrete codes from coarse to fine, predicting all codes within each level in parallel and processing fewer positions at early levels.This yields a short sequence of generation levels rather than full-sequence processing at every step.Source: §2 · pp. 3–4

Discrete latent variables and next scale prediction

• NCM adapts residual quantization and next-scale prediction to text through discrete levels, addressing the lack of a natural image-downsampling counterpart for learning coarse text representations.Accurate reconstruction matters because a single token error can alter meaning, prevent code from running, or invalidate a Sudoku solution.Source: §2 · p. 4

Hierarchical and concept-level text generation

• Prior hierarchical approaches model byte patches, dynamic chunks, sentence representations, or concepts, while NCM represents an entire sequence with discrete levels of increasing resolution.NCM uses a shared transformer to generate levels in order, predicting codes within each level in parallel.Source: §2 · p. 4

NCM’s two-stage training

• Figure 4 distinguishes tokenizer learning and reconstruction from transformer prediction: training predicts all K levels in one pass, while generation predicts levels sequentially from coarse to fine.The transformer predicts codes within a level in parallel from the cumulative reconstruction of earlier levels.Source: Figure 4 · p. 5

3 Next-Concept Modeling

• A multiscale residual VQ-VAE encodes text into progressively finer discrete-code levels, and a transformer generates each level in parallel-conditioned sequence.The tokenizer is frozen before transformer training; the decoder maps the completed reconstruction back to text.Source: §3 · p. 5

3.1 Preliminaries

• Residual quantization uses additional codebooks to represent errors left by earlier quantization, adding corrections to form cumulative partial reconstructions.Multiscale levels use progressively longer code sequences, so short coarse levels precede finer levels that add detail.Source: §3.1 · pp. 5–6

3.2 Formulation

• The generator predicts code levels from coarse to fine, conditioning on earlier levels’ cumulative reconstruction and sampling codes within each level independently.After all levels are generated, the tokenizer decoder maps the cumulative reconstruction to token distributions at all sequence positions.Source: §3.2 · p. 6

3.3 Multiscale Residual Tokenization

• The tokenizer quantizes downsampled residuals at each level, maps selected code vectors to length-L contributions, and adds them to the reconstruction.Nested dropout trains on a sampled prefix of levels to encourage useful early reconstructions; reconstruction cross-entropy and VQ terms are evaluated at that cutoff.Source: §3.3 · pp. 6–7

3.4 Coarse-to-Fine Generation

• With the tokenizer frozen, the transformer is trained on all levels in one forward pass, while generation proceeds coarse to fine in K steps.Conditioning corruption replaces some conditioning codes with codes from the same level’s codebook while preserving ground-truth labels; conditional generation uses a prefix-LM attention pattern.Source: §3.4 · p. 7
• During generation, NCM samples all codes at a level in parallel, adds their reconstruction contribution, then decodes the completed reconstruction to token distributions.The generated token at each position is selected by argmax.Source: §3.4 · p. 7

4.1 Tokenizer Quality

• Tokenizer reconstruction is 100% accurate on Sudoku, while token-level reconstruction on summarization and translation is close to 100%.Across datasets, mean codebook utilization ranges from 62% to 90%; Table 1 reports held-out exact-token recovery and utilization.Source: §4.1 · p. 8, Table 1 · p. 8

4.2 Unconditional Generation

• Across TinyStories, LM1B, and OpenWebText, NCM matches or beats diffusion Gen. PPL with fewer function evaluations while keeping sample entropy close to the data.The figure plots Gen. PPL against function evaluations for TinyStories and LM1B, and against sampling time for OpenWebText.Source: §4.2 · p. 9, Figure 5 · p. 9

4.3 Conditional Generation

• On XSum, unguided NCM beats AURORA-LM on ROUGE-1/2/L with 16× fewer NFE; on WMT14, it trails only E2D2 with 5.8× fewer NFE.With MBR decoding, NCM also beats guided AURORA-LM on all three XSum ROUGE metrics while using fewer NFE.Source: §4.3 · p. 9, Table 2 · p. 10

4.4 Reasoning

• On Sudoku, NCM outperforms autoregressive and diffusion baselines at all three difficulty levels while using fewer function evaluations.The evaluation measures completely solved boards at easy, medium, and hard levels, defined by 40, 35, and 30 clues.Source: §4.4 · p. 10, Table 3 · p. 10

4.5 Scaling

• On LM1B, larger generators improve generation quality and sample diversity at a fixed training-token budget; on XSum, NCM improves over diffusion baselines at matched inference time.The tested LM1B generators have 97M, 317M, and 725M parameters; test-time scaling selects among candidates with MBR decoding.Source: §4.5 · p. 11, Figure 6 · p. 11

4.6 Efficiency

• On LM1B, NCM runs 21× faster than DUO at better generative perplexity; on XSum, it runs 31× faster than AURORA-LM at comparable ROUGE.The speedups reflect both fewer function evaluations and lower average cost per step.Source: §4.6 · pp. 11–12
• On LM1B, tokenizer and generator training use 13.36B token positions, approximately 4.9× fewer than the baselines’ 66B.The budget counts repeated data exposure and includes both tokenizer and generator training.Source: §4.6 · p. 12, Figure 7 · p. 12

4.7 Qualitative Samples

• Qualitative samples cover unconditional LM1B generation, prompted outputs, XSum summarization, and WMT14 German–English translation.The reported conditional outputs remain anchored to their input contexts.Source: Figure 8 · p. 12, §4.7 · p. 12

5 Conclusion

• Across unconditional generation on TinyStories, LM1B, and OpenWebText, NCM achieves competitive or better Gen. PPL than strong diffusion baselines with approximately 4–11× fewer NFE while maintaining sample entropy close to data.On WMT14 translation and XSum summarization, NCM uses approximately 6–16× fewer NFE than unguided diffusion baselines, with competitive translation quality and the best summarization scores; on XSum, it runs 31× faster than the strongest diffusion baseline. On LM1B and OpenWebText, it uses approximately 4–5× fewer training tokens than strong diffusion baselines, including tokenizer and generator training.Source: §5 · p. 13

A Probabilistic Formulation and ELBO

• NCM is formulated as a hierarchical latent-variable model with an evidence lower bound on text log-likelihood, for unconditional generation with deterministic nearest-neighbor code assignments.Source: A Probabilistic Formulation and ELBO · p. 21

A.1 Setup

• The tokenizer deterministically assigns each text sequence a hierarchy of codes, while the decoder defines a text likelihood and the generator a prior over hierarchies.Source: §A.1 · p. 21

A.2 Evidence Lower Bound

• With a frozen tokenizer and decoder and clean conditioning at α = 0, minimizing the generator loss maximizes the ELBO over θ; corruption and nonuniform level weighting modify that objective.The bound uses the complete hierarchy but excludes nested dropout and tokenizer quantization losses.Source: §A.2 · p. 22

• A separate tokenizer and generator are trained for each of the six datasets, grouped by task.Source: B Datasets · p. 22
• Table 4 defines target length as the number of target tokens, equal to the finest-level length ℓK, and reports context lengths in tokens.Source: Table 4 · p. 22
• In Table 4, a dash indicates that the full dataset name is used rather than a separate abbreviation.Source: Table 4 · p. 22

C Architecture Details

• For each dataset, the tokenizer is trained first and frozen while its generator is trained; XSum and WMT14 generators also train first on unpaired text, then on source–target pairs.The target-length and level schedules remain unchanged between those stages.Source: C Architecture Details · p. 22

C.1 Multiscale VQVAE Tokenizer

• The tokenizer appendix reports architectures and training settings; LM1B uses the complete and a truncated hierarchy with respective weights of 0.7 and 0.3.Source: Table 5 · p. 23

C.2 Generator (NCM)

• Generator blocks use self-attention followed by a feed-forward layer, with resized reconstruction and prompt embeddings projected to the hidden width.Attention heads have dimension 64.Source: §C.2 · p. 24

D Baselines

• The baseline appendix documents where Section 4 baseline results come from and how those baselines were sampled.Source: D Baselines · p. 24

LM1B.

• LM1B baselines use Lee et al.’s [15] checkpoints, trained for 1M iterations on 66B tokens, and are sampled at 8, 12, 16, 32, 64, or 128 steps.RDLM is omitted from Figure 5 because its Gen. PPL lies far above the other curves; TinyStories baselines are trained with the NCM generator’s architecture and settings, except FMLM+, which uses a released checkpoint [49].Source: LM1B · pp. 24–25

OpenWebText.

• OpenWebText diffusion baselines use publicly available checkpoints and are sampled with 8 to 1,024 steps.Source: OpenWebText · p. 25

D.2 Conditional Generation

• Conditional baseline quality may come from published results, while timing for baselines without released checkpoints uses randomly initialized weights with matching architectures and task sampling configurations.ELF and AURORA-LM checkpoints are used to reproduce quality results and measure sampling time.Source: §D.2 · p. 25

D.3 Reasoning

• The Sudoku setup reports the percentage of completely solved boards at easy, medium, and hard levels, defined by 40, 35, and 30 clues; baseline results come from Agarwal et al. [49].Source: §D.3 · p. 25

E Additional Results

• The appendix collects unconditional-generation results from Section 4 in addition to the reference-model check.Source: E Additional Results · p. 25

E.1 Generative Perplexity Under Other Reference Models

• NCM has the lowest average Gen. PPL under four additional autoregressive reference models, using approximately 5× fewer training tokens and 12 rather than 128 sampling steps.The evaluation averages five seeds with 256 samples each.Source: §E.1 · p. 25, Table 7 · p. 25
• The additional LM1B evaluation uses the same checkpoints and sampling settings as Figure 5, varying the autoregressive reference model.Table 7 reports generator size, training-token budget, and unweighted average; its caption says baselines use 128 steps and NCM uses 12.Source: §E.1 · p. 25, Table 7 · p. 25

E.2 Unconditional Generation

• Table 8 reports unconditional-generation results for TinyStories, LM1B, and OpenWebText, comparing Gen. PPL and mean per-sample unigram entropy H against ground-truth data.The caption cautions that lower Gen. PPL with entropy below the data, as for FLM and FMLM+ on OpenWebText, partly reflects reduced diversity.Source: §E.2 · p. 26, Table 8 · p. 26

F.1.1 Number of Levels

• Gen. PPL decreases as hierarchy levels increase from K ∈ {4, 8, 16, 32}, with diminishing returns, while all four ladders reconstruct at least 99.95% of tokens.Each tokenizer uses 4,096 codes per level and is trained for 40k steps; its NCM transformer is trained for 10k steps without code corruption. Figure 9 plots Gen. PPL and cumulative reconstruction accuracy.Source: Setup · pp. 26–27, §F.1.1 · p. 26, Figure 9 · p. 27

F.1.2 Level Schedule

• The level schedule controls code growth from 1 to 128 across 12 levels; six schedules are compared, but the supplied text does not identify a winning schedule.Figure 10 shows Gen. PPL over training, final-checkpoint Gen. PPL, and code lengths.Source: §F.1.2 · p. 27, Figure 10 · p. 27

F.2.1 Attention Pattern

• Block-diagonal attention is reported as the best option for low Gen. PPL, fast sampling, and entropy preservation, and is used in NCM.The alternatives are causal, block-causal, and two-level sliding-window attention; block-causal and sliding-window sampling take 1.40× and 1.21× the block-diagonal time, respectively.Source: §F.2.1 · p. 28, Table 9 · p. 28, Figure 11 · p. 28

F.2.2 Context Encoder

• On XSum, frozen T5-small gives slightly higher ROUGE scores than the VQ-VAE encoder, but adds 35.3M parameters—about 40% over the 88M model—and an external pretrained-model dependency.NCM therefore uses its VQ-VAE encoder in reported experiments.Source: §F.2.2 · p. 28, Table 10 · p. 28

F.2.3 Loss Weighting

• Raising loss-weighting α from 0 to 1.5 increases coarse levels’ loss share from 19% to 63%, while Gen. PPL stays at 18.9–19.0 for α ≤ 1 and rises to 19.4 at 1.5.Coarse-level validation loss also falls as α increases.Source: §F.2.3 · p. 29, Figure 12 · p. 29

F.3 Code Corruption

• Moderate code corruption improves generation quality, but excessive corruption weakens conditioning and degrades quality.Training replaces input codes with same-level codebook entries while keeping targets clean; the selected recipe uses a linearly decreasing rate across levels. The authors leave error-based corruption for future work.Source: §F.3 · p. 29, Figure 13 · p. 29

F.4 Sampling

• MBR candidate sampling gains 4.48 ROUGE-1 from K = 1 to K = 256, with diminishing returns.The reported settings yield 29.6 ROUGE-L on XSum at K = 16 and 25.2 BLEU on WMT14 at K = 12; all three ROUGE metrics peak at guidance weight w = 0.5, while length-corrected BLEU peaks at w = 1.0.Source: §F.4 · p. 30, Figure 15 · p. 30, Figure 14 · p. 30

Reconstruction & Utilization

• OpenWebText tokenizer reconstruction accuracy increases across levels, reaching 99.99% at full resolution.Figure 16 plots exact token accuracy after decoding cumulative reconstructions through successive levels.Source: G Tokenizer Analysis · p. 31, Figure 16 · p. 31
• Figure 16 also reports codebook usage by level, showing both the fraction of entries selected at least once and effective utilization measured by code-distribution perplexity as a percentage of codebook size.Source: Figure 16 · p. 31

Tokenizer Failure Cases

• The approximately 0.01% of incorrectly reconstructed OpenWebText tokens include errors in named entities, numerals, diacritics, usernames, markup, and word boundaries.Table 11 presents representative errors by category.Source: G Tokenizer Analysis · p. 31, Table 11 · p. 31

H Probing the Hierarchy

• With the first three levels fixed, samples tend to retain a source story’s broad theme while varying names, objects, and wording.The probes also mix coarse representations from two stories to generate finer levels, and show that replacing two early codes can alter characters, objects, and events while preserving parts of a narrative.Source: H Probing the Hierarchy · pp. 32–34, Figure 17 · p. 32, Figure 18 · p. 33

I Limitations & Future Work

• NCM fixes hierarchy depth and per-level lengths, so even short continuations traverse the full hierarchy; its separately trained tokenizer and roughly 100M-parameter main experiments also bound the reported setting.The authors note occasional artifacts at coarse-level transitions and identify input-dependent schedules, end-to-end training, and larger-scale experiments as future directions.Source: I Limitations & Future Work · p. 35

J Additional Qualitative Samples

• The additional qualitative-sample section includes WMT14, XSum, TinyStories, LM1B, and OpenWebText examples, plus a step-by-step hard Sudoku puzzle.The supplied XSum sample is conditioned on a news context about Willian and Chelsea.Source: J Additional Qualitative Samples · p. 35, §J.3 · p. 36

J.1 WMT14 De–En

• The WMT14 German source example concerns consumer debt in Thailand and the country’s recent exit from a technical recession.Source: §J.1 · p. 35

J.2 Generation Trajectory

• In one translation run, intermediate output quality is not monotonic across levels; the final translation scores 77.8 BLEU.Table 12 shows source, reference, final translation, and level-by-level outputs for that run.Source: §J.2 · p. 36

J.3 XSum

• The supplied XSum example’s context describes Willian’s Chelsea contract, recent goals, and club transfers.Source: §J.3 · p. 36

J.4 TinyStories

• Three TinyStories samples have Gen. PPL values of 10.2, 14.1, and 17.9, with corresponding H values of 3.98, 3.88, and 4.03.Source: §J.4 · p. 37

J.5 LM1B

• Three LM1B samples have Gen. PPL values of 39.4, 66.1, and 72.0, with corresponding H values of 4.29, 4.39, and 4.35.Source: §J.5 · p. 38

J.6 OpenWebText

• Sample 1 has Gen. PPL 47.8 and H 5.40, Sample 2 has PPL 26.4 and H 5.27, and Sample 3 has Gen. PPL 41.5 and H 5.12.The scores are computed on the full samples, although the excerpts are shortened.Source: §J.6 · pp. 38–39
• The displayed OpenWebText excerpts span entertainment and employment topics and contain visibly malformed phrasing.For example, one excerpt starts with “Compise of Doctor,” while another includes “underizedized.”Source: §J.6 · pp. 38–39

J.7 Sudoku

• In the hard Sudoku example, NCM progressively refines the decoded board across hierarchy levels until it reaches the correct solution.Each step generates one level; the figure tracks how many of the 81 cells match the solution.Source: Figure 20 · p. 40