SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

1 INTRODUCTION

• Additional modules or extensive training constrain the applicability of many speculative-decoding methods, while Jacobi-based drafting misaligns with autoregressive pretraining and yields suboptimal acceleration.Jacobi-based methods append pseudo tokens so the target LLM can draft multiple tokens in one decoding step.Source: §1 · p. 1, Figure 1 · p. 2
• Uniform layer skipping can yield 1.2× speedup, but optimal skipped-layer sets are task-specific; transferring a storytelling configuration to reasoning drops speedup from 1.47× to 1.01×.These observations motivate adaptive layer selection during inference rather than using one fixed configuration across tasks.Source: §1 · p. 2
• SWIFT adaptively optimizes skipped target-model layers during inference, using context-based layer selection and confidence-aware acceleration without auxiliary models or additional training.The approach targets diverse input streams and is presented as a plug-and-play form of lossless inference acceleration.Source: §1 · p. 2
• 1.3×–1.6× wall-clock speedup over conventional autoregressive decoding across LLaMA-2 and CodeLLaMA tasks, with the generated-text distribution preserved.In greedy decoding, the reported token acceptance rate across the LLaMA-2 series is 98%–100%.Source: §1 · p. 2
• The introduction frames empirical analysis of layer sparsity and its task-specific nature as groundwork for adaptive self-speculative decoding.The stated contributions include the potential for self-acceleration via layer sparsity and the need for adaptation during inference.Source: §1 · p. 2

2 RELATED WORK

• Autoregressive decoding is memory-bound, and speculative decoding uses a compact draft model whose multiple predictions are validated in parallel by the target LLM.Prior speculative-decoding work includes Xia et al. (2023), Leviathan et al. (2023), and Chen et al. (2023).Source: §2 · p. 2
• Many speculative-decoding variants add modules or training, while Jacobi methods append pseudo tokens; SWIFT complements these approaches with plug-and-play layer-skipping.SWIFT is described as orthogonal to Jacobi-based methods such as Lookahead (Fu et al., 2024).Source: §2 · p. 3, Table 1 · p. 3
• Self-SD [Zhang et al., 2024] and LayerSkip [Elhoushi et al., 2024] skip intermediate layers but require time-consuming offline training, limiting generalization across models and tasks.Other sparsity approaches include quantization, parameter pruning, and knowledge distillation; SWIFT focuses on sparsity within layer computations and investigates lossless acceleration.Source: §2 · p. 3

3.1 SELF-SPECULATIVE DECODING

• Self-SD drafts by skipping intermediate layers, then validates draft tokens with the full model; it optimizes the skipped-layer set to minimize average inference time per token.Bayesian optimization selects a set using training inputs, then holds it fixed during inference.Source: §3.1 · pp. 3–4
• Bayesian optimization constrains Self-SD’s practicality: evaluating eight CNN/Daily Mail samples for 1,000 iterations took nearly 7.5 hours on LLaMA-2-13B and 20 hours on LLaMA-2-70B.The authors say these computational demands restrict generalizability across models and tasks.Source: §3.1 · p. 4

3.2.1 LLMS POSSESS SELF-ACCELERATION POTENTIAL VIA LAYER SPARSITY

• 1.22× average wall-clock speedup over conventional autoregressive decoding, versus Self-SD’s maximum 1.01×, followed a 30% average token-acceptance improvement using top-k drafts.Over 90% of draft tokens were accepted in the reported LLaMA-2-13B experiments.Source: Figure 2 · p. 4, §3.2.1 · p. 4

3.2.2 LAYER SPARSITY IS TASK-SPECIFIC

• 1.47× to 1.01×: applying a storytelling-optimized skipped-layer set to mathematical reasoning reduces speedup; each task’s optimal set differs.The findings motivate adaptive layer optimization for dynamic input streams.Source: Figure 2 · p. 4, §3.2.2 · pp. 4–5

4 SWIFT: ON-THE-FLY SELF-SPECULATIVE DECODING

• SWIFT alternates context-based layer-set optimization with confidence-aware inference acceleration for each input stream.The optimized configuration is used in the acceleration phase.Source: §4 · p. 5, §4.1.2 · p. 6

4.1 CONTEXT-BASED LAYER SET OPTIMIZATION

• On-the-fly optimization balances low overhead and accurate drafting by validating candidate skipped-layer sets against prior LLM-generated tokens in one draft-model forward pass per step.The accumulated context serves as ground truth for evaluating how well each candidate predicts subsequent decoding steps.Source: §4.1 · p. 5

4.1.1 CONTEXT ACCUMULATION

• Uniform layer skipping accelerates generation until the specified context-window length is reached, when inference switches to layer-set optimization.Context accumulation and optimization then alternate until a stopping condition is met.Source: §4.1.1 · p. 5, §4.1.2 · p. 6

4.1.2 LAYER SET OPTIMIZATION

• Candidate layer sets are proposed with random search and periodic Bayesian optimization, then scored by parallel prediction of prior context tokens; the best matchness score guides selection.The optimization phase ends at its step limit or when the best candidate remains unchanged over multiple iterations.Source: §4.1.2 · p. 6

4.2 CONFIDENCE-AWARE INFERENCE ACCELERATION

• During acceleration, the selected draft model proposes tokens autoregressively; confidence controls early stopping and how many candidates are extended for parallel target-model verification.The candidate count is dynamically set from the confidence score, and drafting halts when confidence falls below a threshold.Source: §4.2 · p. 7

5.1 EXPERIMENTAL SETUP

• Experiments use LLaMA-2 and CodeLLaMA across summarization, reasoning, storytelling, and code generation, comparing SWIFT with Parallel and Lookahead Decoding.The evaluation uses CNN/DM, GSM8K, TinyStories, and HumanEval; SWIFT is described as orthogonal to the Jacobi-based baselines.Source: §5.1 · p. 7

5.2 MAIN RESULTS

• SWIFT achieves 1.3×∼1.6× speedups over vanilla autoregressive decoding across models and tasks, with LLaMA-2 token acceptance rates of 98%∼100%.Table 2 compares mean generated length, speedup ratio, and decoding speed under greedy decoding.Source: §5.2 · p. 8, Table 2 · p. 8
• SWIFT achieves 1.3×∼1.5× speedups on code-generation tasks; performance matches autoregressive decoding except for a slight pass@10 variation on CodeLLaMA-34B.Table 3 reports mean generated length, acceptance rate, accuracy, and speedup for greedy pass@1 and sampled pass@10.Source: §5.2 · pp. 8–9, Table 3 · p. 8

5.3 IN-DEPTH ANALYSIS

• During inference, the matchness score rises from 0.45 to 0.73 and then 0.98, after which optimization stops; acceleration-phase instance speedup averages 1.53×.The overall speedup rises as more tokens are generated, approaching the average instance speedup.Source: §5.3 · p. 9
• On 1,000 CNN/DM samples with LLaMA-2-13B, optimization takes 0.8% of inference and reduces optimization time nearly 180× versus Self-SD.Self-SD's cited optimization time is 7.5 hours; SWIFT's drafting stage consumes most inference latency.Source: §5.3 · p. 9
• Across concatenated task-by-task input streams, SWIFT maintains 96% average acceptance and 1.3×∼1.6× speedups, while Self-SD falls to 68% acceptance and 1.05× speedup under shifts.The evaluation spans summarization, reasoning, instruction following, translation, and question answering.Source: Figure 7 · p. 9, §5.3 · p. 10
• Reducing maximum optimization steps S can hasten acceleration for short inputs, while increasing Bayesian interval β reduces early optimization overhead; larger models show higher optimal skip ratios and speedups.The authors identify acceleration of models such as 175B as future investigation.Source: §5.3 · p. 10, Figure 8 · p. 10
• On Yi-34B and DeepSeek-Coder-33B and their instruction-tuned variants, SWIFT reports efficiency improvements of 26% to 54%.These backbones are evaluated on text and code generation tasks, respectively.Source: §5.3 · p. 10

• SWIFT adaptively selects intermediate LLM layers to skip during inference and requires neither additional training nor auxiliary models.The conclusion describes it as a plug-and-play solution across diverse input streams.Source: §6 · p. 10
• Experiments report 1.3×∼1.6× speedups while preserving the distribution of generated text.The conclusion also highlights performance on dynamic input streams and integration with varied LLM backbones.Source: §6 · p. 10

• The datasets are publicly released, human-labeled in English, and contain no personal information.The statement says user privacy is protected during dataset creation.Source: ETHICS STATEMENT · p. 11

• Replication materials include code, environment configurations, scripts, and relevant experimental settings.The settings cover models, datasets, inference setup, and evaluation metrics.Source: REPRODUCIBILITY STATEMENT · p. 11

A PRELIMINARY DETAILS

• Self-SD’s optimal skipped-layer configurations vary across four task domains.The visualization covers LLaMA-2-13B.Source: A PRELIMINARY DETAILS · p. 17
• In Figure 10, gray marks retained layers, red skipped attention layers, and blue skipped MLP layers.Source: Figure 10 · p. 17

B.1 MODELS AND DATASETS

• Evaluation covers LLaMA-2 and CodeLLaMA on summarization, mathematical reasoning, storytelling, code generation, instruction following, translation, and question answering.Datasets include CNN/DM, GSM8K, TinyStories, HumanEval, Alpaca, WMT14 DE-EN, and Natural Questions.Source: §B.1 · p. 17

B.2 INFERENCE SETUP

• Experiments use random search and Bayesian optimization, with early stopping after 300 unimproved steps or when matchness exceeds 0.95.The maximum optimization steps are 1,000, with Bayesian optimization every 25 steps.Source: §B.2 · p. 17

B.3 EVALUATION METRICS

• M is average target-LLM output tokens per forward pass, α is the accepted-token fraction, and expected speedup depends on M, α, and cost coefficient c.The cost coefficient is the draft-model single-forward time divided by the target-LLM single-forward time; larger M and α and smaller c correspond to higher ideal speedup.Source: §B.3 · p. 18

C.1 DETAILS OF MAIN RESULTS

• α exceeds 90% across evaluation settings, while M ranges from 2.99 to 8.21 in the reported main results.Table 4 reports M, α, and expected speedup under greedy decoding with FP16 precision.Source: Table 4 · p. 18, §C.1 · p. 18

C.2 ADDITIONAL RESULTS ON LLAMA-70B MODELS

• SWIFT achieves 1.4×–1.5× wall-clock speedup across LLaMA-2 and LLaMA-3 70B models, with α above 85% and M from 3.43 to 7.80.The skip ratio differs between model versions, yet acceleration remains consistent.Source: §C.2 · pp. 18–19, Table 5 · p. 19

C.3 DETAILED RESULTS OF LLM BACKBONES

• SWIFT achieves 1.2×–1.3× wall-clock speedup on Yi-34B and 1.3×–1.5× on DeepSeek-Coder-33B.On DeepSeek-Coder-33B, M ranges from 3.16 to 4.17 and α exceeds 83%.Source: §C.3 · p. 19, Table 6 · p. 19, Table 7 · p. 20

D.1 ABLATION STUDY

• Early-stopping drafting improves α by 55%, and layer-set optimization improves speedup by 34% over initial uniform layer skipping.Removing dynamic verification decreases both M and overall speedup.Source: §D.1 · pp. 19–20, Table 8 · p. 20

D.2 CONTEXT WINDOW

• γ = 32 balances optimization performance and computational overhead; smaller windows increase matchness randomness, while larger ones increase optimization cost.Table 9 reports speedups and optimization-step latency across context-window settings.Source: Table 9 · p. 20, §D.2 · p. 20

D.3 COMPARISONS WITH PRIOR LAYER-SKIPPING METHODS

• SWIFT preserves the target distribution and achieves 1.56× speedup without training; Self-SD’s optimization latency is reduced by approximately 200×.LayerSkip’s pretraining or fine-tuning alters the target distribution; the comparison reports skip ratio, M, α, and speedup under greedy decoding.Source: §D.3 · pp. 20–21, Table 10 · p. 21, Table 11 · p. 21

D.4 DETAILED COMPARISONS WITH SELF-SD

• At similar optimization latencies, SWIFT reaches 1.56× versus 0.97× speedup (116s versus 155s); Self-SD at 100 iterations reaches 1.19× with 2898s versus SWIFT’s 116s.The comparisons use LLaMA-2-13B on CNN/DM under greedy decoding.Source: §D.4 · p. 21, Figure 11 · p. 21, Table 12 · p. 22

D.5 THE NECESSITY OF PLUG-AND-PLAY SD METHODS

• Plug-and-play SD is model-agnostic and training-free, useful for large models and rapid integration; SWIFT reports 1.3×–1.6× speedups across models and tasks.The paper also describes SWIFT as orthogonal to Lookahead, whose combination could further amplify efficiency.Source: §D.5 · p. 23

D.6 ADDITIONAL DISCUSSIONS WITH RELATED WORK

• SWIFT complements training-required drafting approaches by using layer skipping without auxiliary models or additional training, while preserving the target model’s original output distribution.Source: §D.6 · p. 23

D.7 OPTIMIZATION STEPS

• Across optimization steps, SWIFT’s skipped-layer set is gradually refined toward the optimal configuration.Figure 12 visualizes layer configurations at different optimization steps.Source: §D.7 · p. 23, Figure 12 · p. 24

D.8 CASE STUDY

• In CNN/DM and HumanEval examples, SWIFT accepts more drafts than Lookahead, with accepted tokens distributed uniformly in outputs.Tables 13 and 14 show examples for LLaMA-2-13B and CodeLLaMA-13B, respectively.Source: §D.8 · p. 24, Table 13 · p. 24, Table 14 · p. 24