SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

1 Introduction

• A separate, well-aligned drafter complicates deployment, motivating self-speculative decoding within a unified model.Speculative decoding amortizes the cost of token generation by verifying multiple proposed tokens in one target-model forward pass.Source: §1 · p. 1
• Early-exit drafting is cheap but sacrifices draft quality by discarding deep representations, whereas MTP preserves depth but requires a full forward pass at every drafting step.On GSM8K with draft length 4, SEED's throughput is approximately 2.0× early-exit and 1.6× MTP.Source: §1 · p. 1, Figure 1 · p. 2
• SEED drafts with a lightweight decoder conditioned on deep contextual representations cached during verification, avoiding repeated encoder computation.The paper presents this as an implicit encoder-decoder view of the standard decoder-only transformer.Source: §1 · p. 2
• Up to 2.7× average speedup on 4B-scale models, with generation quality matching or improving standard autoregressive fine-tuning.SEED also outperforms early-exit and MTP self-speculative baselines and runs 28% faster than EAGLE-3.Source: §1 · p. 2

2 Related Work

• Self-speculative methods draft from the target model itself and broadly use multi-token prediction or early exit by skipping layers.Classical speculative decoding instead uses a small auxiliary model or multiple heads.Source: §2 · pp. 2–3
• MTP predicts multiple tokens per forward pass, but pretraining-from-scratch variants are impractical for deployed models, while alternative Jacobi or diffusion schemes can sacrifice quality or complicate training.More recent work adapts MTP for faster inference on existing models.Source: §2 · p. 3
• Early-exit methods draft from initial layers; SEED instead skips those layers and uses final layers conditioned on full-depth representations.The paper describes this inverted layer-skipping design as producing higher-quality drafts than early-exit alternatives.Source: §2 · p. 3

3 Preliminaries

• A transformer maps an input token sequence to a probability distribution over the next token and consists of layers that process hidden states from preceding layers.The first layer operates on token embeddings, and the final layer's output is projected back to vocabulary space.Source: §3 · p. 3
• Each self-attention layer computes keys and values that are cached to avoid redundant computation; the cache available before a token is processed is denoted KV at that layer.The passage notes that cache updates are omitted from the subsequent method description for brevity.Source: §3 · p. 3
• Autoregressive transformer language models are trained by minimizing standard next-token cross-entropy over the training data distribution.The distribution is denoted D.Source: §3 · p. 3

4.1 Method

• SEED conceptually partitions a decoder-only transformer into an encoder that builds deep prefix representations and a thin decoder that emits the next token; this decomposition is equivalent to standard autoregressive decoding.The encoder typically contains most layers, such as 26 of 28 in the reported experiments.Source: §4.1 · p. 4
• Verification and encoding merge: the full model verifies proposed drafts and caches deep representations of the verified prefix for the next drafting round.Between verifications, the lightweight decoder drafts from raw token embeddings while attending to the cached representations; the encoder runs again at the next verification.Source: §4.1 · p. 4

4.2 Training Algorithm

• Training combines standard autoregressive loss with a speculative drafting loss that teaches the decoder to predict from raw embeddings and cached deep representations.Block-causal attention makes the decoder's training context resemble drafting between verification steps, and blocks can be processed in parallel.Source: §4.2 · pp. 5–6

4.3 Inference Algorithm

• At inference, the decoder drafts a fixed number of tokens for full-model verification, or uses confidence-based stopping and sibling candidates verified as a tree.The adaptive strategy drafts more when confident and fewer when uncertain; the decoder's confidence reflects acceptance rate because it shares parameters with the verifier.Source: §4.3 · p. 6

5.1 Experimental Setup

• Evaluation covers GSM8K, KodCode, ScienceQA, and CNN/Daily Mail, using throughput plus pass@1 or ROUGE and comparing AR, self-speculative, separate-drafter, diffusion, and MTP baselines.The study fine-tunes Qwen3-1.7B-Base and Qwen3-4B-Base; SEED uses the last two layers as decoder and training block size 10.Source: §5.1 · pp. 7–8

5.2 Results

• Across all benchmarks, SEED averages 2.6× speedup on Qwen3-1.7B-Base and 2.7× on Qwen3-4B-Base, outperforming self-speculative baselines and matching or surpassing EAGLE-3.It also improves task performance over standard autoregressive fine-tuning.Source: §5.2 · p. 8

5.3 Analysis

• SEED beats LayerSkip and EAGLE-3 in draft acceptance through full-depth representations and verifier-shared layers, while matching MTP draft quality at lower cost.SEED reuses representations cached during verification and needs only a 2-layer decoder pass per draft token.Source: §5.3 · pp. 8–9, Figure 3 · p. 8
• Linear probes find SEED encoder representations more predictive of distant future tokens than standard AR, consistent with a planning-ahead effect.The probe compares prediction accuracy across lookahead steps.Source: Figure 4 · p. 8, §5.3 · p. 9
• On GSM8K, SEED stays similar to AR across early and middle layers but diverges most near layer 26; LayerSkip shows broader representational drift.The authors suggest SEED’s localized changes preserve deep contextual processing in most of the model.Source: §5.3 · p. 9, Figure 5 · p. 9

5.4 Ablation Study

• Larger decoders raise draft acceptance but lower throughput; larger training blocks generally raise acceptance and throughput, and λ=1.0 balances draft quality with task accuracy.The main experiments use a 2-layer decoder and block size 10.Source: §5.4 · pp. 9–10, Figure 6 · p. 10, Figure 7 · p. 10

• SEED reuses cached deep representations of the verified prefix, with the model’s last layers drafting while earlier layers provide contextual features.Source: §6 · p. 10
• Across diverse benchmarks, SEED consistently outperforms prior self-speculative baselines in decoding speed while preserving or often improving task performance relative to standard autoregressive fine-tuning.Source: §6 · p. 10
• SEED’s gains are associated with training dynamics near the encoder-decoder interface that likely encourage more future-predictive internal representations.Source: §6 · p. 10

A Limitations and Future Directions

• Experiments cover Qwen3-1.7B-Base and Qwen3-4B-Base, so SEED’s efficiency-quality tradeoff has not been validated at larger model scales.The authors identify evaluation on 8B-, 14B-, or 32B-scale models as future work.Source: A Limitations and Future Directions · p. 14
• The study focuses mainly on task-specific supervised fine-tuning; extending SEED to broader post-training or pretraining remains future work.The authors propose studying whether planning-ahead behavior improves general capabilities and robustness beyond the task-specific benchmarks.Source: A Limitations and Future Directions · p. 14
• The main experiments use a fixed split with the top two layers as decoder, while optimal partitioning and realized speedup may vary with models, tasks, sequences, and deployment hardware.The authors point to adaptive partitioning and hardware-aware kernels as future directions.Source: A Limitations and Future Directions · p. 14

B Additional Discussions on Related Work

• SEED preserves the standard next-token objective and is described as integrating into existing supervised fine-tuning pipelines without external modules.The authors contrast this with methods that use training data for speedup often at the expense of generation quality.Source: B Additional Discussions on Related Work · p. 14

B.1 Connection to Self-Speculative Decoding with Layer Skipping

• Layer-skipping methods often retain early layers, whose KV caches encode shallow prefix representations; SEED instead drafts from final layers conditioned on deep verified-prefix caches.The authors state this design preserves representational strength while keeping draft computation inexpensive, enabling higher acceptance rates and throughput.Source: §B.1 · p. 15

B.2 Comparison to the EAGLE Series

• Unlike EAGLE’s feature-level autoregression, SEED directly predicts tokens from raw draft embeddings while attending to deep cached representations, avoiding its feature-prediction distribution shift.SEED’s shared drafter-verifier parameters also keep it better aligned with the target distribution, according to the authors.Source: §B.2 · pp. 15–16

B.3 Comparison to E2D2

• SEED achieves much better task performance than E2D2 while using a smaller decoder in the controlled Qwen3-1.7B-Base GSM8K comparison.Source: Table 4 · p. 16, §B.3 · p. 16
• Throughput is nearly identical at every split, but E2D2 trades accuracy for speed while SEED accuracy stays essentially flat across decoder sizes because exact verification preserves the autoregressive output distribution.The comparison matches training budget, block size b = 4, and encoder-decoder split, and uses fixed draft length 4.Source: §B.3 · p. 16

C.1 Hardware

• The 1.7B models were fine-tuned on one 48 GB NVIDIA A6000 GPU, while 4B models used H100 or B200 GPUs; all models were evaluated on one A6000.Source: §C.1 · p. 17

C.2 Datasets

• The task-specific benchmarks are GSM8K for math reasoning, KodCode for code generation, ScienceQA for general knowledge, and CNN/Daily Mail for summarization.The described dataset setup includes task-specific preprocessing and evaluation subsets or length constraints.Source: §C.2 · p. 17

C.3 Baselines

• Baseline implementations use method-specific settings, including a four-layer decoder for E2D2, eight draft tokens for Apple MTP except four for summarization, and released or adapted implementations for other baselines.EAGLE-3 evaluation uses instruct-model checkpoints paired with Qwen3 base-model names because corresponding base-model weights were unavailable.Source: §C.3 · pp. 17–18

C.4 Training

• Training uses the same optimization setup across methods unless noted, with a 100-step linear warm-up, cosine decay, validation-loss checkpoint selection, and seed 1.Hyperparameters vary by model scale and dataset, primarily in learning rate, batch size, and maximum sequence length.Source: §C.4 · p. 18

C.5 Task Prompts

• The prompts include step-by-step math instructions, programming-task wording, multiple-choice framing with answer options, and an article summarization instruction.Source: Task Prompts · p. 19

C.6 Evaluation

• Throughput evaluation terminates generation at the end-of-text token to avoid counting repetitive outputs produced when models are forced to continue.Source: §C.6 · p. 19

D.1 Algorithms

• SEED’s adaptive procedure varies candidate branch width with decoder confidence, then verifies packed tree candidates in one full-model pass using a tree-structured attention mask.Candidates attend to the verified prefix and their own ancestral path; accepted tokens update the prefix and KV cache for the next drafting round.Source: §D.1 · pp. 21–22, Figure 8 · p. 21

D.2 Further Ablations

• Across τ values from 0.5 to 0.9, decoding throughput remains stable even as higher thresholds improve acceptance rate but shorten acceptance length.The experiments use a GSM8K-finetuned Qwen3-1.7B-Base SEED model; τ = 0.7 is selected generally, and τ = 0.5 for CNN/Daily Mail summarization.Source: §D.2 · p. 22, Figure 9 · p. 22
• Tree-based verification consistently improves throughput, acceptance length, and acceptance rate under fixed and dynamic drafting on GSM8K.The ablation uses a GSM8K-finetuned Qwen3-1.7B-Base SEED model.Source: Table 6 · p. 22, §D.2 · p. 23
• Removing parameter sharing substantially degrades both draft quality and task performance, despite retaining deep-feature conditioning and having more parameters.The independent drafter also has poorer token acceptance and reduced throughput; the authors attribute an alignment benefit to parameter sharing.Source: §D.2 · p. 23, Table 7 · p. 23

E.1 Why SEED Improves Task Performance

• MTP-style training predicts multiple future tokens from one contextual representation, encouraging that representation to encode information about tokens beyond the next one.The passage describes this as implicitly planning ahead.Source: §E.1 · p. 23
• In SEED, speculative-loss gradients primarily pass through cached contextual representations because raw token embeddings are fixed lookup vectors.The encoder is therefore encouraged to compress upcoming-token information into those representations for the decoder’s drafts.Source: §E.1 · p. 23
• The authors train linear probes on intermediate representations from SEED and autoregressively fine-tuned models to predict future tokens on GSM8K.Both models use block size 4; probes target lookahead steps 1–5 using 20,000 target-only token positions from 1,000 test samples.Source: §E.1 · pp. 23–24
• Figure 10 reports training linear probes to predict future tokens from intermediate representations on Kodcode; its supplied caption identifies the probe task and dataset but provides no comparative outcomes.Source: Figure 10 · p. 24

E.2 Accelerating an Already Well-Tuned Model

• Retrofitting SEED onto an AR-fine-tuned model preserves GSM8K task accuracy while achieving essentially the same decoding throughput as SEED trained from the base model.The table caption describes continued fine-tuning with dynamic drafting at τ = 0.7 and tree-based verification, and reports comparable speedup.Source: §E.2 · p. 24, Table 8 · p. 24
• A small accuracy gain after retrofitting is consistent with the MTP-like “planning ahead” effect induced by the speculative objective.The authors’ CKA analysis indicates that SEED mainly reshapes representations near the encoder–decoder interface while leaving earlier layers largely unchanged, which they say suggests compatibility with standard AR fine-tuning.Source: §E.2 · p. 24

F Computational Cost Analysis

• The cost model is a parameter-based estimate that ignores operation-specific FLOPs, memory-access patterns, and hardware utilization.It closely predicts SEED’s measured throughput for fixed drafting without tree-based verification, while discrepancies for LayerSkip and Apple MTP are attributed to omitted costs and implementation overheads.Source: F Computational Cost Analysis · p. 25
• SEED’s drafting step is 4.2× cheaper than a full pass and 1.7× cheaper than LayerSkip’s early-exit pass, despite using deeper representations.The analysis is an approximate parameter-based estimate for a 1.7B model.Source: F Computational Cost Analysis · p. 25, Table 9 · p. 25
• SEED’s advantage over EAGLE-3 does not primarily come from lower absolute drafting cost, but from improved draft quality and higher acceptance rates.EAGLE-3 has a lower per-token drafting cost of 0.08 units.Source: F Computational Cost Analysis · pp. 25–26
• The relative drafting cost of SEED falls from 0.239 units on Qwen3-1.7B-Base to 0.147 units on Qwen3-4B-Base.The analysis links this scaling behavior to the larger average speedup observed on the 4B model.Source: F Computational Cost Analysis · p. 26

G Broader Impacts

• More efficient LLM inference may reduce deployment cost, latency, and energy use, while potentially making models more accessible.The authors also warn that lower generation cost and latency could amplify risks such as large-scale harmful or misleading content.Source: G Broader Impacts · p. 26
• Efficiency-related misuse risks are not unique to SEED; the authors call for evaluating whether acceleration changes their scale or detectability.The concern is relevant because SEED improves LLM-generation efficiency.Source: G Broader Impacts · p. 26