Source-linked AI summary
BS-Spec: BFP-Based Bit-Sharing for Self-Speculative LLM Decoding with LUT-Driven Mixed-Precision Acceleration
Jiwoo Kim, Gunho Park, Sunwoo Yoo, Baeseong Park, Dongsoo Lee, Youngjoo Lee
TL;DR
Existing self-speculative methods can rely on fixed configurations even though workload conditions vary, while separate draft models add memory overhead. BS-Spec derives draft representations from one BFP target model and selects workload-specific precision pairs; its reported hardware breakdown finds only marginal area and power contributions from the BFP sign decoder.
Problem
Existing self-speculative approaches often use fixed draft configurations and optimize weight and KV-cache costs in isolation, despite serving workloads spanning different batch sizes and sequence lengths.
Source: §I · p. 1Method
BS-Spec derives lower-precision drafts through MSB-only partial loading from a shared-exponent BFP target representation and uses offline calibration with a precision lookup table to select workload-specific weight and KV-cache precisions.
Source: §I · pp. 1–2Results
The BS-RAC sign decoder contributes only a marginal fraction of total area and power, indicating negligible hardware overhead for BFP support.
Source: C. BS-RAC Power and Area Breakdown · p. 11Takeaways & Limitations
BS-Spec combines bit sharing, workload-aware precision selection, and BFP hardware execution to support variable-precision draft and target computation.
Source: §I · p. 2, §VI · p. 11
Abstract
from arXiv · showhide
As large language models (LLMs) are increasingly deployed across diverse inference services, serving workloads span a wide range of batch sizes, context lengths, and performance requirements. Speculative decoding improves inference throughput, but conventional approaches require a separate draft model and KV cache, while existing self-speculative methods suffer from workload-insensitive draft configurations and inefficient mixed-precision execution. In this paper, we present BS-Spec, a block floating point (BFP)-based bit-sharing framework that jointly optimizes draft construction, workload-aware precision selection, and hardware execution. BS-Spec derives multiple draft representations directly from a single BFP target model by partially loading the most significant bits of weights and the KV cache while preserving shared exponents, eliminating separate draft-model storage. Offline calibration selects the weight and KV-cache precision pair that maximizes estimated speedup for each serving regime, and a precision lookup table applies the selected configuration at runtime. To support this adaptive execution, we design a mixed-precision BFP accelerator with a lookup table (LUT)-based variable-precision datapath for both draft and target computation. Experimental results show that BS-Spec achieves up to $1.9\times$ speedup over conventional autoregressive decoding and improves speedup over the evaluated bit-sharing self-speculative decoding baselines by $67.2\%$ on average, demonstrating the benefits of jointly co-designing bit sharing, precision adaptation, and specialized hardware for diverse LLM serving workloads.
I. INTRODUCTION
BS-Spec addresses workload-sensitive draft efficiency by deriving drafts from a single BFP target representation and adapting weight and KV-cache precision. It pairs offline precision selection with hardware designed for variable-precision BFP execution.
I. INTRODUCTION
- Existing quantization-based self-speculative methods use fixed configurations and optimize weight and KV-cache costs separately, limiting efficiency across diverse serving workloads.The tradeoff between memory-access cost and draft acceptance varies with batch size and sequence length.Source: §I · pp. 1–2
- BS-Spec derives multiple draft precisions by partially loading the most significant bits of BFP target weights and KV cache under shared exponents, avoiding separate draft-model storage and conversion.The target path remains unchanged for verification.Source: §I · pp. 1–2
- Offline calibration jointly selects weight and KV-cache precision based on draft cost and acceptance rate, then stores configurations in a precision lookup table for runtime workload-based selection.The selected pair is intended to maximize speculative-decoding speedup for the serving workload.Source: §I · pp. 1–2
- A bit-serial mixed-precision accelerator executes BFP weights and KV cache with BF16 activations, using LUT-based accumulation and a decoding-oriented dataflow for draft and target computation.The design supports runtime-variable BFP5–BFP9 precision.Source: §I · p. 2
II. BACKGROUND
Autoregressive decoding repeatedly accesses model weights and a growing KV cache, with the relative memory costs varying across serving workloads. Speculative decoding trades draft-generation cost against verification cost and the expected number of accepted tokens.
A. Memory Bottlenecks in Autoregressive LLM Decoding
- Decoding repeatedly accesses model weights and the growing KV cache, while prefill is characterized by parallel processing of input tokens and large batched matrix multiplications.The passage distinguishes these phases by their performance characteristics.Source: A. Memory Bottlenecks in… · p. 2
- Weight traffic dominates shorter-context decoding, while KV-cache traffic grows in importance at longer contexts and larger batch sizes, making isolated optimization workload-limited.The relative costs of the two traffic sources vary across serving workloads.Source: A. Memory Bottlenecks in… · p. 3
B. Speculative Decoding and the Draft Cost-Acceptance Tradeoff
- Speculative decoding drafts multiple candidate tokens for target verification; its speedup depends on verification-to-target cost, draft-to-target cost, and expected accepted length.Expected accepted length depends on draft length and acceptance rate.Source: B. Speculative Decoding and the Draft… · p. 3
- The best draft configuration can vary by serving regime because workload changes alter the relative weight and KV-cache traffic contributions and the draft-cost-to-acceptance tradeoff.Reducing draft memory traffic lowers draft cost, while preserving draft-target alignment supports acceptance.Source: B. Speculative Decoding and the Draft… · p. 3
C. Self-Speculative Decoding and the Need for Bit-Sharing Co-Design
BS-Spec is presented as a way to achieve speedup across evaluated workload regions by jointly adapting weight and KV-cache precision. Its hardware design targets the flexible, bit-shared execution that this approach requires.
C. Self-Speculative Decoding and the Need for Bit-Sharing Co-Design
- Self-speculative decoding derives draft predictions from the target model, avoiding storage of a separate draft model.The draft becomes an implicit low-cost representation of the target.Source: C. Self-Speculative Decoding and the… · p. 3
- BS-Spec achieves speedup across all evaluated workload regions, motivating joint adaptation of weight and KV-cache precision.The section contrasts this broad coverage with a QuantSpec variant whose benefit diminishes in long-sequence, large-batch regimes.Source: C. Self-Speculative Decoding and the… · p. 4
D. Mixed-Precision Hardware for Bit-Shared Self Speculative Decoding
- Bit-shared decoding reuses target-model bits for draft execution, requiring one hardware platform to support draft and near-full-precision target computation.The hardware must efficiently load selected bits and compute with them within a shared framework.Source: D. Mixed-Precision Hardware for… · p. 4
- BS-Spec combines bit-serial load-and-compute with LUT-based accumulation to support fine-grained bit control while mitigating serialized-operation overhead.The section identifies existing architectures as poorly matched to bit-shared speculative decoding’s flexible precision and shared-bit reuse requirements.Source: D. Mixed-Precision Hardware for… · p. 4
III. BS-SPEC ALGORITHM
BS-Spec uses a BFP target representation to derive lower-precision drafts through partial-bit loading. Shared exponents preserve numerical closeness while allowing draft generation to reduce memory traffic.
A. BFP Target Representation
- BFP groups aligned significands under a shared exponent, supporting draft construction directly from the target representation.The paper identifies hardware-friendly construction and numerical closeness to the target as advantages of this representation.Source: A. BFP Target Representation · pp. 4–5
- In the 7-bit partial-loading example, BFP retains the closest approximation to the original value among the compared schemes.The example’s group size of three makes exponent overhead appear relatively large; a group size such as 64 amortizes it across more values.Source: A. BFP Target Representation · p. 5
- Loading only a subset of target-model bits reduces draft-phase memory traffic, and BFP supports multiple draft precisions through partial-bit loading.This approach is described as reducing draft cost while maintaining high acceptance rate.Source: A. BFP Target Representation · p. 5
- BS-Spec stores weights in BFP9 and quantizes KV-cache values online, grouping keys channel-wise and values token-wise.Keys are buffered until a full channel-wise group forms; activations remain in BF16.Source: A. BFP Target Representation · p. 5
B. Self-Speculative Decoding with BFP
BS-Spec derives draft paths from one BFP target representation and independently selects weight and KV-cache precision. The selection balances draft cost and acceptance rate, which vary with workload.
B. Self-Speculative Decoding with BFP
- BS-Spec creates draft tokens from reduced-precision weight and KV-cache bits, then verifies candidates with the full BFP representation and commits only accepted tokens.The draft phase creates temporary KV entries for the drafted sequence.Source: B. Self-Speculative Decoding with BFP · p. 5
- Independent selection of weight and KV-cache precision lets BS-Spec instantiate different draft paths from the same stored representation.This avoids tying the method to a single fixed draft model.Source: B. Self-Speculative Decoding with BFP · p. 5
C. Trade-Offs in Draft Model Precision
- Weight footprint stays fixed while KV-cache footprint grows with batch size and sequence length, so their memory-traffic contributions—and suitable precisions—vary by workload.The section describes comparing precision combinations across batch sizes and sequence lengths to guide joint selection.Source: C. Trade-Offs in Draft Model Precision · p. 6, Figure 6 · p. 6
- Lower draft cost generally reduces acceptance rate, while a larger draft-to-target ratio typically improves acceptance at greater draft cost.BS-Spec estimates speedup using draft cost and measured acceptance rate; γ = 7 is selected for the largest expected accepted generation length on average.Source: C. Trade-Offs in Draft Model Precision · p. 6
D. Runtime Draft Precision Selection for the Best Speedup
BS-Spec estimates workload-specific speedup, stores the best weight and KV-cache precision pair in a lookup table, and applies that choice at runtime. Its bit-plane-major datapath translates precision choices into reduced memory transfer and computation.
D. Runtime Draft Precision Selection for the Best Speedup
- Under the targeted memory-bound regime, draft-to-target memory-loading ratio approximates the latency ratio used in BS-Spec’s speedup estimate.The weight and KV-cache loading ratios are functions of batch size and sequence length.Source: D. Runtime Draft Precision Selection… · p. 6
- Offline calibration selects the weight and KV-cache precision pair with the highest estimated speedup at each workload point and stores it in the PLT for runtime retrieval.The selected configuration is mapped directly onto BS-Spec hardware.Source: D. Runtime Draft Precision Selection… · p. 7, Table V · p. 7
A. Variable-Precision Bit-Plane Execution
- Bit-plane-major storage lets the controller fetch selected MSB planes for drafting and extend bit-serial execution across remaining planes for target verification.Draft and target computation share a representation, memory organization, and datapath; precision changes by changing the number of loaded and executed planes.Source: A. Variable-Precision Bit-Plane… · p. 7, Figure 8 · p. 7
- Reducing weight or KV-cache precision decreases both off-chip bit-plane transfers and bit-serial accumulation iterations.The precision selected by the algorithm therefore reduces data movement and execution cost directly.Source: A. Variable-Precision Bit-Plane… · p. 7
B. LUT-driven dataflow
BS-Spec uses LUT-based accumulation to compute products between BF16 activations and bit-serial BFP weights or KV-cache values. It replaces multiplication with indexed LUT access and accumulation, with serial bit-plane processing setting execution cycles.
B. LUT-driven dataflow
- LUT-based accumulation replaces multiplication with indexed lookup of precomputed partial sums for binary weight patterns, followed by accumulation.Activation groups are used to precompute sums for possible weight patterns.Source: B. LUT-driven dataflow · pp. 7–8
- Weight bit planes flow through the datapath and are broadcast across engines; the KV cache follows the same dataflow.The figure also depicts the desired computation, exponent-modified activation, and LUT indices with their corresponding activation sums.Source: Figure 10 · p. 8, Figure 9 · p. 8
- Streamed weight bits form LUT indices, and selected entries are shifted by bit position before local accumulation.LUT construction is pipelined with execution to hide most generation latency.Source: B. LUT-driven dataflow · p. 8
C. BFP-Aware LUT Computation
BS-Spec folds BFP shared-exponent alignment into LUT construction, then uses a decoding-oriented dataflow and integrated system architecture to execute variable-precision computations.
C. BFP-Aware LUT Computation
- BFP shared-exponent alignment is folded into LUT precomputation by scaling activations before constructing entries for aligned activation mantissas.The resulting computation uses LUT lookup followed by integer accumulation; sign is handled separately from magnitude bit planes.Source: C. BFP-Aware LUT Computation · p. 8
D. Decoding-Oriented Dataflow
- The decoding-oriented dataflow streams weight bit planes as LUT indices while each MPU keeps LUT entries and partial sums local for concurrent output tiles.Weights are broadcast across MPUs, whereas LUT entries and partial sums remain private to each MPU.Source: D. Decoding-Oriented Dataflow · p. 8
E. System Integration
- The integrated system combines a host, global controller, memory subsystem, and compute core, with DMA transfers overlapping execution.The core includes buffers, multiple MPUs, and a special function unit for post-processing.Source: E. System Integration · pp. 8–9, Figure 11 · p. 8
V. EXPERIMENTAL RESULTS
BS-Spec is evaluated against SPEQ and QuantSpec using normalized throughput and energy, with cycle-level modeling under shared memory-bandwidth constraints. Across the evaluated models and workloads, it achieves the highest overall efficiency, while its measured throughput and energy vary by batch size and model.
A. Hardware Evaluation Setup
- The evaluation models all architectures with the same memory hierarchy and bandwidth constraints and compares BS-Spec with SPEQ and QuantSpec.QuantSpec is mapped to a synthesized multi-precision systolic array; reported latency and energy include computation, SRAM and HBM accesses, and pipeline overlap.Source: A. Hardware Evaluation Setup · p. 9, B. Speedup and Energy Evaluation · p. 9
B. Speedup and Energy Evaluation
- The normalized throughput and per-token energy use conventional autoregressive BF16 target decoding on a systolic-array architecture as the baseline.The evaluated architectures are configured to match normalized output throughput.Source: B. Speedup and Energy Evaluation · p. 9, A. Hardware Evaluation Setup · p. 9
- 1.83×, 1.84×, and 1.90× speedups at batch size 4, and 1.62×, 1.68×, and 1.92× at batch size 32, are reported for BS-Spec across the three evaluated models.The reported model order is Llama-3.2-3B, Llama-3.1-8B, and Qwen3-14B; Fig. 12 compares normalized throughput against target-only autoregressive decoding.Source: B. Speedup and Energy Evaluation · p. 10
- 0.55×, 0.54×, and 0.53× baseline energy at batch size 4, and 0.55×, 0.52×, and 0.43× at batch size 32, are reported for BS-Spec across the three evaluated models.The model order is Llama-3.2-3B, Llama-3.1-8B, and Qwen3-14B; Fig. 13 compares normalized energy.Source: B. Speedup and Energy Evaluation · p. 10, Figure 13 · p. 10
- BS-Spec achieves the highest overall efficiency across short- and long-context regimes by adapting weight and KV-cache precisions while retaining one bit-sharing method and datapath.At batch size 4, SPEQ is more effective for short contexts, while QuantSpec is relatively more competitive for long contexts; at batch size 32, higher arithmetic intensity reduces memory traffic's impact.Source: B. Speedup and Energy Evaluation · p. 10, Figure 14 · p. 10
C. BS-RAC Power and Area Breakdown
The BS-RAC unit replaces multiplication with selection logic, and its sign decoder adds only a marginal fraction of area and power.
C. BS-RAC Power and Area Breakdown
- The sign decoder contributes only a marginal fraction of BS-RAC area and power, indicating negligible hardware overhead for BFP support.BS-RAC extends the FIGLUT RAC structure with a lightweight sign decoder for sign-magnitude BFP operands.Source: C. BS-RAC Power and Area Breakdown · p. 11
D. Ablation Study
The ablation finds that BS-Spec’s proposed dataflow sustains speedup across sequence lengths, with speculative decoding contributing the largest gain. The conclusion describes workload-adaptive draft precision and a BFP accelerator as parts of the framework.
D. Ablation Study
- Draft-phase HBM accesses account for the largest share of energy and latency, although γ draft executions remain considerably cheaper than γ additional target-model decoding steps.The breakdown includes γ draft executions and one target execution.Source: D. Ablation Study · p. 11
- Speculative decoding delivers the largest improvement by replacing multiple target steps with low-cost draft executions and one verification; LUT-stationary hardware adds acceleration over systolic arrays.BFP9 target compression provides a more modest gain by reducing target execution cost.Source: Figure 17 · p. 11, D. Ablation Study · p. 11
- The proposed dataflow maintains approximately 2× speedup across sequence lengths, while FIGLUT-style mapping degrades as sequence length and target precision increase.The comparison covers draft and target linear and attention operations on Qwen3-14B at batch size 4.Source: Figure 18 · p. 11, D. Ablation Study · p. 11
VI. CONCLUSION
- BS-Spec derives draft precisions through MSB-only partial loading, selects workload-specific weight and KV-cache precisions offline, and applies them at runtime through a PLT.The conclusion also describes a bit-serial BFP accelerator with BFP-aware LUT generation and LUT-stationary dataflow for variable-precision draft and target computation.Source: §VI · p. 11