PaperlayerView exact revision on arXiv

Source-linked AI summary

BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration

Chence Yang, Ningxi Cheng, Arash Akbari, Qitao Tan, Qingchan Zhu, Ci Zhang, Changdi Yang, Yanzhi Wang, Wei Niu, Jinhui Wang, Jin Lu, Geng Yuan

arXiv:2610.02800v1cs.AI

TL;DR

Speculative decoding can reduce expensive token-by-token target execution, but extra draft-model storage is costly on memory-constrained devices. BitNest nests a low-precision draft within a refined higher-precision representation and extends the design to the KV cache. Across evaluated models and workloads, it reports 95.2% average acceptance and 1.48–1.61× speedup over 16-bit autoregressive decoding.

  • Problem

    Additional draft models or representations consume scarce memory, while self-speculative approaches still trade off draft quality, target quality, and storage efficiency.

    Source: §1 · p. 1

  • Method

    BitNest constructs a strong low-precision base first, adds refinement for the higher-precision target, and extends nested precision access to the KV cache.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    Across four 7B–8B models and six workloads, BitNest achieves 95.2% average speculative acceptance and 1.48–1.61× speedup over 16-bit autoregressive decoding while closely preserving higher-precision target quality.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    A single physical representation supports low-precision drafting and higher-precision verification without maintaining multiple model copies.

    Source: §1 · p. 2

    BitNest focuses on self-speculative decoding and does not currently exploit auxiliary drafting architectures that can provide greater candidate-generation parallelism.

    Source: §6 · p. 9

Abstract

from arXiv · show

Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However, existing methods often require an additional draft model or weight representation, introducing non-negligible memory overhead on resource-constrained devices. Self-speculative approaches reduce this overhead, yet still face trade-offs between draft quality, target quality, and storage efficiency. We propose BitNest, a bit-nested speculative decoding framework that embeds a low-precision draft directly into the higher-precision target representation. Instead of deriving a draft from a predefined target, BitNest first constructs a strong low-precision base and then recovers the higher-precision target through residual refinement, enabling both models to share a single physical weight representation. BitNest further extends this progressive-precision design to the KV cache for long-context inference. Across multiple 7B--8B edge-friendly LLMs and diverse workloads, BitNest achieves an average speculative acceptance rate of 95.2% while closely preserving higher-precision model quality, and delivers 1.48--1.61x end-to-end speedup over FP16 autoregressive decoding. On the LLaMA models supported by all representative self-speculative baselines, BitNest also achieves consistently competitive or higher decoding speedup.

1 INTRODUCTION

BitNest addresses the memory cost of speculative decoding by nesting a low-precision draft within a higher-precision target representation. It constructs the draft first, then refines it to recover the target, and reports strong acceptance and decoding speedups across evaluated models and workloads.

1 INTRODUCTION

  • Additional draft models or representations compete with the target model and KV cache for scarce memory on resource-constrained devices.Under FP16 inference, a 7B model alone can approach a 16 GB device’s capacity, leaving little room for KV cache and runtime buffers.Source: §1 · pp. 1–2
  • Existing self-speculative designs trade drafting efficiency, target quality, or storage, motivating a single representation that supports a strong draft and higher-quality target.Layer-skipping methods couple drafting cost with prediction quality; quantization-based approaches either constrain the target to shared low-bit weights or retain additional draft weights.Source: §1 · p. 2
  • BitNest builds a strong low-precision base first, then adds residual refinement so the draft and higher-precision target share one physical representation.Drafting reads only the base; target verification reads the refinement as well. The approach also extends nesting to the KV cache.Source: §1 · pp. 2–3
  • 1.48–1.61× speedup over 16-bit autoregressive decoding across evaluated models accompanies 95.2% average speculative acceptance across four models and six workloads.BitNest closely preserves higher-precision target quality; on Jetson Orin NX 16GB, it achieves approximately 1.5× speedup on LLaMA-2-7B-32K and extends maximum executable context from 6K to 10K.Source: §1 · pp. 2–3

2 RELATED WORK

Related work develops speculative decoding through improved candidate generation, cheaper self-speculative execution paths, and low-bit quantization. BitNest is distinguished by nesting draft and target precision views within one physical representation.

2.1 SPECULATIVE DECODING

  • Speculative decoding methods improve candidate generation through multi-token prediction, feature-space drafting, and parallel diffusion-based drafting, but auxiliary drafters add parameters or representations and memory overhead.DFlash uses a lightweight block-diffusion drafter to generate candidate blocks in parallel.Source: §2.1 · p. 3

2.2 SELF-SPECULATIVE DECODING

  • Self-speculative methods avoid a separately deployed draft model, but layer-skipping approaches couple drafting efficiency to prediction quality, while quantization methods either constrain target precision or retain extra draft weights.BitNest instead embeds the draft within the higher-precision target representation.Source: §2.2 · p. 3

2.3 LLM QUANTIZATION AND ROTATION-BASED PTQ

  • Rotation-based PTQ methods use function-preserving transformations to improve robustness to low-precision quantization, whereas BitNest organizes multiple precision views as a nested representation.The cited methods primarily optimize individual quantized configurations rather than explicitly nesting them.Source: §2.3 · pp. 3–4

3 METHODOLOGY

BitNest represents its draft and target as nested precision views, then makes those views independently accessible for drafting, verification, and long-context KV-cache use. Its base-first construction preserves the draft while refinement recovers the higher-precision target.

3.1 OVERVIEW

  • BitNest transforms weights into an equivalent representation, encodes a base and residual refinement, and nests the draft within the higher-precision target.The implementation uses W4 for drafting and W8 for verification, with corresponding KV4 drafting and KV8 verification views.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 BITNEST WEIGHT CONSTRUCTION

  • The W4 base is optimized first and fixed; encoding its remaining quantization error as refinement recovers the W8 target without modifying the draft.The W4 draft is an exact subset of the W8 target representation.Source: §3.2 · pp. 4–5

3.3 DUAL-PLANE WEIGHT STORAGE

  • Dual-plane storage keeps base and refinement independently accessible, so drafting fetches only base codes while verification reads both planes to reconstruct W8.Each plane occupies N/2 bytes for N weights; the refinement scale is derived from the base scale without additional scale metadata.Source: Figure 3 · p. 5, §3.3 · p. 5

3.4 CONTEXT-AWARE KV NESTING

  • For long contexts, the draft reads the KV base while verification reads the full cache; accepted entries are retained and the rejected suffix discarded.BitNest uses KV8 for both stages at short contexts, switching to KV4 drafting and KV8 verification at long contexts to reduce KV-cache traffic.Source: §3.4 · pp. 5–6, Figure 4 · p. 6

4 EXPERIMENT

Experiments covered four LLMs, six workloads, and a single RTX A6000 GPU, with additional long-context and Jetson Orin NX evaluations. BitNest combines decoding speedups with target-quality preservation, reduced memory use, and edge-device gains.

4.2 SPEEDUP FOR LLM INFERENCE

  • 1.48–1.61× end-to-end speedup over the corresponding 16-bit autoregressive baseline across four models, with gains across workloads.BitNest also achieved 1.35× decode speedup over FP16 autoregressive decoding on OpenVLA-7B while preserving task-level performance.Source: Table 1 · p. 6, §4.2 · pp. 6–7
  • 95.2% average generation-time acceptance accompanied quality close to FP16 and standalone W8A8 across evaluated models and workloads.On LLaMA-3-8B, BitNest achieved 49.8% GSM8K accuracy versus 50.0% for W8A8.Source: §4.2 · p. 7

4.3 SPEEDUP AT LONG CONTEXT LENGTHS

  • At 32K context, drafting from the KV4 base achieved 1.66× speedup, while full-KV8 BitNest declined from 1.57× at 4K to 1.38× at 32K.Figure 5 compares decoding speedup and peak GPU memory across context lengths.Source: Figure 5 · p. 7, §4.3 · p. 8

4.4 MEMORY SAVING

  • At 32K context, BitNest used 21.0 GiB peak memory, versus 24.0 GiB for QSpec and 25.1 GiB for QuantSpec.On LLaMA-2-7B, nested weights occupied 6.22 GiB, compared with 9.38 GiB for an independent W4 draft plus INT8 target.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 EDGE-DEVICE EVALUATION

  • On Jetson Orin NX, BitNest achieved approximately 1.5× speedup on LLaMA-2-7B-32K and extended executable context from 2K with FP16 to 10K.On the 16GB device, decode energy fell from 1.52 to 0.95 J/token relative to FP16; on the 8GB device, Llama-3.2-3B reached up to 1.57× speedup and energy fell from 0.768 to 0.46 J/token.Source: Table 4 · p. 8, §4.5 · p. 8

5 ABLATION STUDY

The ablations show that low-precision representations can draft effectively but may not preserve final-output fidelity. Retaining higher precision for verification supports accuracy, while speculation length affects throughput.

5.1 EFFECT OF SPECULATION LENGTH

  • At speculation length γ = 4, LLaMA-2-7B-32K throughput peaked at 59.3 tok/s at 4K and 34.3 tok/s at 31K.Larger γ further reduced acceptance without improving throughput.Source: §5.1 · p. 9

5.3 WHY RETAIN W8 FOR THE TARGET?

  • On LLaMA-3-8B, W4A8 achieved 41.4% GSM8K accuracy versus 50.0% for W8A8, while BitNest refinement recovered accuracy to 49.8%.W4A16 reached 41.8%, indicating that the dominant degradation came from 4-bit weights rather than activation quantization.Source: Table 5 · p. 9, §5.3 · p. 9

5.2 WHY RETAIN KV8 FOR THE TARGET?

  • On Qwen2.5, reducing target KV precision from KV8 to KV4 lowered GSM8K accuracy from 81.6% to 78.4%, despite 0.933 speculative acceptance.The design therefore uses KV4 for drafting and retains KV8 for verification.Source: §5.2 · p. 9

6 LIMITATION

BitNest focuses on self-speculative decoding and does not currently use auxiliary drafting architectures that can provide greater candidate-generation parallelism.

6 LIMITATION

  • BitNest does not currently exploit auxiliary drafting architectures; DFlash may achieve stronger acceleration when additional draft-model overhead is acceptable.Supporting auxiliary drafters while retaining nested-representation memory benefits is identified as future work.Source: §6 · p. 9

AI USE STATEMENT

The authors used generative AI tools for experimental implementation and code, language editing, and manuscript organization, and report reviewing and testing AI-assisted work.

AI USE STATEMENT

  • Authors report that they reviewed and tested AI-assisted implementation and verified the resulting experimental data for correctness and consistency.They state that AI tools were not used to develop the theoretical or conceptual framework, formulate hypotheses or mathematical claims, or design the methodology or experiments.Source: AI USE STATEMENT · p. 10

A.1 DETAILED BIT-NESTED WEIGHT CONSTRUCTION

BitNest constructs a low-precision W4 base from transformed pretrained weights, then adds residual refinement to recover an 8-bit target while preserving the draft.

A.1 DETAILED BIT-NESTED WEIGHT CONSTRUCTION

  • BitNest folds function-preserving R1/R2 transformations into pretrained weights before constructing the W4 base with GPTQ and group size 128.Source: §A.1 · p. 13
  • Draft weights are reconstructed from the 4-bit codes q4 and group-wise scale s4.Source: §A.1 · p. 13
  • After quantization, q4 and s4 remain fixed while residual quantization refines the base to recover the 8-bit target without modifying the W4 draft.Source: §A.1 · p. 13
  • An asymmetric constraint for q4 = −8 keeps the composed integer code within the signed 8-bit range.Source: §A.1 · p. 13

A.2 DETAILED KV-CACHE NESTING

BitNest nests KV4 drafting and KV8 verification representations in one stored cache entry, using online per-token, per-head quantization and context-dependent access policies.

A.2 DETAILED KV-CACHE NESTING

  • On cache writes, BitNest applies online per-token, per-KV-head symmetric round-to-nearest quantization to vectors with head dimension d = 128.Source: §A.2 · p. 13
  • The KV4 drafting representation uses the low-precision base, while an additional 4-bit refinement with scale s/16 yields the KV8 target.Source: §A.2 · pp. 13–14
  • Each KV entry is stored once as nested base and refinement planes and can be accessed as either KV4 or KV8.Source: §A.2 · p. 14
  • During drafting, historical context uses KV4; verification reads both planes, recomputes proposed-token states, retains accepted entries, and discards the rejected suffix.Source: §A.2 · p. 14
  • BitNest uses KV8 for both phases at short contexts and KV4 drafting with KV8 verification at long contexts; Recent-N offers an intermediate fidelity–traffic trade-off.Source: §A.2 · p. 14

A.3 EXPERIMENTAL SETUP

The appendix details BitNest’s evaluation and ablations, including its speculative agreement, precision trade-offs, and edge-device performance. Results examine where lower-precision drafting helps and where implementation or task conditions limit the benefit.

A.3 EXPERIMENTAL SETUP

  • The LLM evaluation covers four model families and six workloads, uses native baseline configurations, and measures quality, throughput, speedup, memory, and acceptance.Main experiments use a single RTX A6000 at batch size 1; long-context experiments use PG19 prefixes from 4K to 32K tokens.Source: §A.3 · p. 14

A.4 EXPERIMENT ON VISION-LANGUAGE-ACTION MODEL

  • On LIBERO Spatial, BitNest reaches 89% success and 1.35× decode-phase speedup, but prefill overhead leaves per-action latency at 212.7 ms versus 201.5 ms for FP16 AR.This was an exploratory VLA evaluation, not a comparison with recent VLA architectures.Source: §A.4 · p. 15, Table 6 · p. 15, Table 7 · p. 15

A.5.1 DRAFT–TARGET AGREEMENT

  • Across four models and six workloads, BitNest averages 95.2% acceptance while using W4A8 drafting and nested W8A8 verification.Model-wise average acceptance ranges from 94.8% to 95.7%.Source: §A.5.1 · pp. 15–16, Table 8 · p. 16

A.5.3 EFFECT OF DRAFT KV PRECISION

  • At 32K, draft KV precision raises LLaMA-2-7B-32K throughput from 23.4 tok/s with BF16 to 34.3 tok/s with KV4, while Qwen2.5-7B rises from 50.7 to 51.4 tok/s.For LLaMA-2, KV4 has lower acceptance than KV8 (0.875 vs. 0.944); γ = 4 gives the highest throughput in both tested settings.Source: §A.5.3 · pp. 16–17, Table 10 · p. 16, §A.5.2 · p. 16

A.5.4 ADDITIONAL TARGET-PRECISION ANALYSIS

  • On Qwen2.5-7B, KV4 verification lowers GSM8K accuracy to 78.4% from 82.4% with KV8 autoregressive decoding, while KV4 drafting alone retains 81.6%.Average LongBench scores remain relatively stable across these configurations; on LLaMA-2-7B-32K, KV4 drafting preserves the KV8 target’s 45.7 average, while KV4 for both drafting and verification yields 45.3.Source: §A.5.4 · p. 17, Table 11 · p. 17
  • On LLaMA-3-8B, BitNest reaches 49.8% GSM8K accuracy versus 41.4% for W4A8 and 50.0% for W8A8; on Qwen2.5-7B, it reaches 82.0% versus 79.3% and 83.2%.The results support the low-precision base for drafting and refinement for recovering final-prediction fidelity.Source: §A.5.4 · pp. 17–18, Table 12 · p. 17

A.6 ADDITIONAL EDGE-DEVICE RESULTS

  • On Orin NX16, BitNest reaches 9.0–9.3 tok/s vs 6.0 tok/s for FP16 AR on LLaMA-2-7B-32K and extends executable context from 2K to 10K.For Qwen2.5-7B, FP16 exceeds device memory at 2K context, while BitNest remains executable. The optimized GEMV kernel raises BitNest throughput on LLaMA-2-7B-32K from 7.4–7.9 to 9.0–9.3 tok/s.Source: §A.6.1 · p. 18, §A.6.2 · p. 18, Table 13 · p. 18

Related papers

Loading 2610.02800v1…