Next Token or Vector? Learning How Far to Decode

1 Introduction

• Token-by-token decoding makes latency grow with response length, while predictable and uncertain continuations call for different generation granularity.The introduction connects slower decoding to increased user wait time and reduced serving capacity under fixed computational resources.Source: §1 · p. 1
• The introduction asks whether horizon selection among latent span representations can move the draft-length decision before decoding the proposed span.It situates this question against methods that vary draft length through learned stopping or dynamic tree construction.Source: §1 · pp. 1–2
• CADENCE uses horizon-specific Gaussian span representations and a learned router to select a draft length or fall back to the backbone token.A shared decoder converts the selected Gaussian mean into parallel token predictions, which the backbone verifies before continuing from the accepted prefix.Source: §1 · p. 2
• The experiments compare router-selected with fixed-horizon generation, test whether Gaussian parameters improve routing, and measure throughput against greedy decoding.The supplied introduction states these evaluation questions but does not report their outcomes.Source: §1 · p. 2

2 Related Work

• The related-work discussion organizes prior methods by generation process, predicted representation, and draft-length control.Table 1 summarizes distinctions among the closest methods.Source: §2 · p. 2

2.1 Diffusion and Insertion Models

• Diffusion and insertion approaches change the generation process, whereas CADENCE retains causal backbone decoding and proposes variable-length continuations from the accepted prefix.The passage contrasts reverse discrete diffusion, block diffusion, and insertion prediction with CADENCE’s approach.Source: §2.1 · p. 2

2.2 Continuous Representations

• CALM and CoLaR feed compressed or Gaussian latent representations into subsequent backbone steps, while CADENCE decodes a selected span latent into parallel token predictions.CADENCE continues from the prefix accepted by the backbone.Source: §2.2 · p. 2

2.3 Speculative and Adaptive Decoding

• Speculative methods vary drafting or verification, while CADENCE selects a horizon-specific Gaussian span latent before decoding and verifies the resulting draft’s accepted prefix.Figure 2 also identifies its backbone context, shared decoder, router, and training objectives: Gaussian NLL, KL self-distillation, and full-span router supervision.Source: §2.3 · p. 2, Figure 2 · p. 3

3 Methodology

• For each step, the backbone predicts the next token and supplies context for horizon-specific Gaussian heads; a router selects a horizon or backbone-only fallback.The shared decoder predicts k additional tokens in parallel, and greedy verification commits the longest matching prefix, with accepted length 1 ≤ a ≤ k + 1.Source: §3 · p. 3

3.1 Multi-step Gaussian heads

• The context representation concatenates the last-token states from the backbone’s final three layers and projects them to backbone width.This gives downstream components a common input width without increasing their parameter cost through concatenation.Source: §3.1 · p. 3
• Each horizon has a separate head that predicts a D-dimensional diagonal Gaussian, while a shared MLP trunk processes the context.The shared trunk captures information across span lengths; the separate heads learn horizon-specific latent distributions, using D = 512 for all horizons.Source: §3.1 · p. 3
• A learned attention pooler compresses embeddings of the k future tokens into one span target, using rotary position embeddings on queries and keys.The span begins at t+2 because the backbone predicts t+1 directly; the pooler is pretrained with the decoder and then frozen to fix the Gaussian-head target.Source: §3.1 · p. 4
• Each Gaussian head is trained by minimizing the negative log-likelihood of its pooled span target.The modeling choice represents the k future tokens as a continuous diagonal Gaussian distribution.Source: §3.1 · pp. 3–4

3.2 Perceiver IO Decoder

• A shared Perceiver-style decoder uses the selected span latent with positional queries to produce k token distributions in parallel across horizons.Sharing decoder parameters and positional queries encourages compatible latent representations and avoids a separate decoder for each span length.Source: §3.2 · p. 4

3.3 Router

• The router conditions horizon selection on backbone context and Gaussian parameters for candidate spans, choosing among five horizons or a backbone-only fallback.Its six output classes comprise the five prediction horizons and the fallback.Source: §3.3 · p. 4

3.4 Training

• Training proceeds in three stages: latent reconstruction, contextual span prediction, and router training with the span-prediction modules frozen.The schedule keeps draft behavior fixed while the router learns its selection policy.Source: §3.4 · p. 4

3.4.1 Decoder and pooler pretraining

• Pooler and decoder pretraining reconstructs ground-truth span embeddings before Gaussian heads predict points in the latent space.The pooler and decoder are trained together, then the pooler is frozen to fix the targets for Gaussian negative log-likelihood training.Source: §3.4.1 · p. 4

3.4.2 Joint training of the Gaussian heads

• Joint training combines Gaussian negative log-likelihood for pooled span targets with token-level self-distillation from the backbone.Distillation uses backbone distributions also used in greedy verification; the NLL is downweighted because it sums over 512 latent dimensions and five horizons.Source: §3.4.2 · p. 5

3.4.3 Router training

• Router labels mark the largest horizon whose entire predicted span matches backbone targets, or class 0 when none matches.Because longer horizons yield fewer full-span matches, training uses focal loss and inverse-square-root class-frequency weights to increase the contribution of rare long-horizon labels.Source: §3.4.3 · p. 5

• The selected Gaussian mean supplies the decoder latent; the proposal includes the backbone token followed by k decoder tokens, for k + 1 total.The backbone-only fallback commits just the backbone token, and proposed drafts undergo greedy verification before accepted tokens are committed.Source: §3.5 · p. 5

4.1 Implementation Details

• The frozen Llama-3.1-8B-Instruct backbone uses five 512-dimensional Gaussian heads and a residual MLP router to select generation horizons.The context representation concatenates last-token hidden states from the final three backbone layers; a shared Perceiver-style decoder predicts future tokens.Source: §4.1 · pp. 5–6

4.2 Datasets

• Training uses UltraChat-ShareGPT, while evaluation covers Alpaca, GSM8K, and HumanEval for free-form generation, mathematical reasoning, and code generation.The training data is generated with Llama-3.1-8B-Instruct.Source: §4.2 · p. 6

4.3 Evaluation Metrics

• Evaluation measures token and full-draft acceptance, mean accepted length per step, and throughput speedup against autoregressive decoding with the same backbone.Figure 3 compares token acceptance by horizon for router-selected positions and fixed-horizon generation under greedy verification.Source: §4.3 · p. 6, Figure 3 · p. 6

5.1 Effectiveness of Adaptive Step-Size Selection

• At k = 4, router-selected token acceptance is 51.7% versus 37.7% on GSM8K, 64.3% versus 35.8% on HumanEval, and 60.5% versus 36.3% on Alpaca.Acceptance generally decreases as the horizon grows, but router-selected positions outperform fixed-horizon generation at the corresponding horizon; the comparison is conditional on selected positions.Source: §5.1 · pp. 6–7

5.2 Generation Granularity Across Tasks

• Single-token steps remain most frequent, but CADENCE also advances multiple tokens across Alpaca, GSM8K, and HumanEval.Examples show longer accepted steps in templates, prompt copies, and code syntax, and shorter steps around sentence openings, content-specific words, and numerical answers.Source: §5.2 · p. 7, Figure 5 · p. 7, Figure 4 · p. 7

5.3 Effectiveness of Gaussian-Enhanced Router

• With Gaussian parameters, k = 4 token acceptance rises from 56.9% to 60.5%, and full-draft acceptance rises from 21.5% to 31.3%.At k = 5, mean accepted length increases from 3.27 to 3.57 tokens; overall multi-token attempts rise from 47.1% to 50.0%, and overall mean accepted length from 1.50 to 1.56 tokens.Source: Table 2 · p. 8, §5.3 · p. 8

5.4 Generation Efficiency

• Throughput speedups over greedy Llama-3.1-8B-Instruct are 1.28× on Alpaca, 1.28× on GSM8K, and 1.3× on HumanEval.The comparison uses the same backbone; the reported generation preserves the backbone’s outputs.Source: Table 3 · p. 8, §5.4 · p. 8

6 Conclusion

• CADENCE combines Gaussian future-span representations with learned horizon selection to vary the number of tokens predicted at each decoding step.Source: §6 · p. 8
• Across instruction following, mathematical reasoning, and code generation, longer accepted continuations frequently occur in structured and predictable spans.Source: §6 · p. 8
• Under greedy verification, CADENCE preserves the backbone’s generated outputs while providing additional inference-efficiency benefits.Source: §6 · p. 8

7 Limitations

• The router can choose shorter horizons or backbone-only generation when longer continuations could be accepted.The authors identify this as a conservative routing behavior.Source: §7 · p. 8
• Experiments use only the Llama-3.1-8B-Instruct backbone, so other backbones are outside the evaluated scope.Source: §7 · p. 8
• CADENCE relies on backbone verification, and its behavior without verification is not systematically evaluated.Source: §7 · p. 8

A.1 Exact-match agreement of router

• The UltraChat router is evaluated against exact-match labels at every token position across 80 Alpaca greedy answers (17,359 positions), where horizon k is the backbone token plus k learned tokens.Class 0 means no horizon matches in full.Source: §A.1 · p. 10, Table 4 · p. 10
• 59.1% of positions have exact router-label agreement, versus 43.0% for an always-no-match predictor.The router is within one class of the label on 85.2% of positions; 25.4% of disagreements are shorter than the label and 15.5% are longer.Source: Table 4 · p. 10, §A.1 · p. 10
• Agreement on no-match is 35.0% and on k=1 is 13.8%; the largest miss is router no-match when the label is k=1, at 12.3%.Horizons past k=2 are rare both as predictions and as labels.Source: §A.1 · p. 10
• Agreement is lower in the first answer tokens, where longer full matches are more common and the router still abstains.Further into answers, no-match labels become more common and agreement rises with them.Source: §A.1 · p. 10