Accelerating CPU Speculative Decoding via Unified Graph Execution and Transactional KV Isolation

1 Introduction

• Separate draft and target contexts require memory swaps, thread-pool switching, and graph reconstruction that introduce synchronization latency.These execution costs can negate speculative decoding’s theoretical speedups.Source: §1 · p. 1
• Rejected proposals can make KV-cache rollback require temporary buffers, page-table remapping, or cache-block copies, adding memory traffic.Source: §1 · p. 1
• SubTernary combines Unified Graph Caching and Transactional KV-Cache Isolation, with systems context and cache management delivering +35.0% of throughput gains versus +6.0% from its ternary kernel.The 256-bit AVX2 ternary kernel is presented as a supporting low-bit case study.Source: §1 · p. 1

2.1 Unified Graph Caching

• SubTernary places separate static draft and target graphs in one memory context, sharing memory arenas and a persistent thread pool to avoid dynamic allocation, context swaps, and graph reconstruction.All CPU threads are assigned sequentially to the active graph: draft first, then target.Source: §2.1 · p. 2
• 8.73 t/s versus 4.17 t/s: sequential allocation of all CPU threads outperforms a 4/4 core-partitioned configuration by 2.09×.Partitioning leaves one model’s cores idle during the other’s phase and can add context switching and synchronization.Source: §2.1 · p. 2

2.2 Transactional KV-Cache Isolation

• A static scratchpad at the end of the verified cache holds speculative activations; causal masking hides stale entries, and rollback resets the KV write pointer instead of copying memory.Accepted tokens commit in place by advancing the pointer.Source: §2.2 · p. 3

3 Hardware-Algorithm Co-Design

• A custom 256-bit AVX2 SIMD kernel executes ternary weights W ∈ {−1, 0, 1} by packing them into binary sign and sparsity masks for register additions.Source: §3 · p. 3
• Quantization-aware training with a straight-through estimator distills draft models to prevent vocabulary representation collapse.Source: §3 · p. 3
• The distilled ternary draft maintains ≥70% token acceptance, and the runtime is described as supporting low-bit drafting without overhead erasing algorithmic gains.Source: §3 · p. 3

4.1 Experimental Setup

• Experiments use an 8-vCPU environment pinned to eight physical cores on an Intel Xeon Gold 5218 host, with five independent evaluation trials.The target is Llama-3.2-3B-Instruct and the draft is Llama-3.2-1B-Instruct; metrics are reported as mean ± standard deviation.Source: §4.1 · p. 3

4.2 Component Ablation and Context Scaling

• 3.99× overall speedup from systems contributions alone; unified graph caching (+1.23x stepwise) and transactional KV isolation (+1.09x stepwise) account for +35.0% throughput gains over native llama.cpp speculative decoding.Table 1 reports an incremental component ablation for Llama-3.2 3B Q8_0 target / 1B draft at context 2048.Source: Table 1 · p. 3, §4.2 · p. 3
• At L = 8192, SubTernary sustains 0.874 t/s (2.96x speedup over llama.cpp spec) with 3.74 ms rollback latency.At that context length, llama.cpp rollback latency reaches 24.69 ms and throughput falls to 0.295 t/s, below the AR baseline of 0.556 t/s.Source: §4.2 · pp. 3–4, Table 2 · p. 4

4.3 Microarchitectural Efficiency and Resilience Under Contention

• SubTernary reduces DRAM memory bandwidth consumption to 8.1 GB/s and raises instructions per cycle (IPC) to 2.34 in microarchitectural profiling.Table 3 profiles a 2048 × 2048 matrix over 50 iterations.Source: §4.3 · p. 4, Table 3 · p. 4
• Under heavy shared hypervisor load, SubTernary maintains positive throughput scaling while llama.cpp speculative decoding falls below autoregressive speeds.Small proposal batches (K ≤2) suffer synchronization overhead, while K ≥4 amortizes latency; Figure 2 plots latency against proposal length.Source: §4.3 · p. 4, Figure 2 · p. 4

• SubTernary addresses context and cache bottlenecks in CPU speculative decoding through Unified Graph Caching and Transactional KV Isolation.Source: §5 · p. 4
• SubTernary achieves up to a 3.99x speedup on commodity CPUs.Source: §5 · p. 4
• SubTernary scales robustly.Source: §5 · p. 4

A Limitations and Scope of Hardware Evaluation

• Evaluation is limited to an Intel Xeon Gold 5218 server; broader microarchitecture and multi-socket NUMA evaluation remains future work.The host represents bandwidth-limited, AVX2-capable commodity x86_64 processors; performance dynamics may differ on AMD Zen, Intel Sapphire Rapids, or ARM Neoverse.Source: A Limitations and Scope of Hardware… · p. 6

B Related Work

• Prior low-bit CPU work uses LUTs or ternary weights, while direct PTQ on sub-3B drafts yields a 0.0% acceptance rate; SubTernary instead uses QAT distillation.The passage discusses T-MAC, BitNet [Ma et al., 2024], and QuantSpec [Tiwari et al., 2024].Source: B Related Work · p. 6
• Related KV-cache methods target tree verification, self-speculation, or GPU page management, while Medusa and EAGLE use parallel heads or recurrent draft trees.The passage contrasts these approaches with SubTernary's static scratchpad rollbacks and single-sequence graph execution on commodity CPUs.Source: B Related Work · p. 6

C Complete Experimental Setup and Reproducibility Environment

• Experiments use an eight-vCPU Xeon Gold 5218 setup, Llama and Qwen target/draft models, greedy decoding with K = 4, and five independent trials.The environment uses Ubuntu 24.04.1, GCC 13.3.0, and eight pinned threads; reported metrics are means ± standard deviations.Source: C Complete Experimental Setup and… · pp. 6–7

D.1 RoPE Positional Alignment Under Partial Acceptance

• After accepting m tokens, SubTernary commits the KV pointer at t + m and overwrites later scratchpad entries without reindexing or moving data.Preassigned sequential RoPE position IDs keep the accepted keys and values aligned.Source: §D.1 · p. 7

D.2.1 Analytical Proof of Distribution Exactness

• The target model’s verification pass reproduces the autoregressive distribution because it uses target weights and masks rejected scratchpad regions.The analytical argument says LayerNorm, RoPE, and intermediate activations follow the target model’s forward path.Source: §D.2.1 · p. 7

D.2.2 KL-Divergence Empirical Validation

• Across temperatures 0.0–1.5, measured KL divergence supports identical output distributions and zero algorithmic drift.Table 4 reports divergence bounded by floating-point precision.Source: Table 4 · p. 8, §D.2.2 · p. 8

D.3 Scaled Downstream Task Quality Sweeps

• 100.00% exact token matches against the autoregressive baseline were maintained across HumanEval and GSM8K, with zero score degradation.Table 5 reports the scaled downstream task quality sweep.Source: Table 5 · p. 8, §D.3 · p. 8

E Architectural Divergence: CPU vs. GPU Speculators

• 4.17× and 42.63× speedups over Medusa and EAGLE, respectively, were reported for raw step latency on commodity hardware.The comparison benchmarks CPU implementations on Llama-3.2-3B-Instruct.Source: Table 6 · p. 8, E Architectural Divergence: CPU vs.… · p. 8

F.1 Clean vs. Throttled CPU Performance Sweeps

• Under heavy hypervisor noise, llama.cpp speculative decoding falls to a 0.27× penalty, while SubTernary maintains positive scaling.The unified memory and persistent thread pool reduce synchronization boundaries under OS jitter.Source: Table 8 · p. 9, §F.1 · p. 9

F.2 Baseline CPU Engine Latency Verification

• 35.00±0.42 ms/token is reported for GGML Q4_0, which the paper says outperforms PyTorch Compiled heavily.The paper uses this comparison to validate its CPU baseline; Table 9 reports the latency verification.Source: Table 9 · p. 9, §F.2 · p. 9

F.3 Out-of-Domain Governor Validation Across Domains and Temperatures

• When acceptance falls below 30% on out-of-domain text, the governor suspends drafting and reverts to autoregressive generation.The paper also reports that the protection works at extreme sampling temperatures when α = 0.0%.Source: §F.3 · p. 9, Table 10 · p. 10, Table 11 · p. 10

F.4 Target Model Scaling Sweep (3B vs. 8B)

• SubTernary’s speedup factor rises from 1.70× on the 3B target to 2.80× on the 8B target.The scaling sweep uses FP16 target models; the paper relates the increase to the larger target memory footprint.Source: Table 12 · p. 10, §F.4 · p. 10

F.5 Per-Layer Quantization Sensitivity Analysis

• Quantizing attention weights to {−1, 0, 1} causes acceptance rates to collapse to 0.00% in the evaluated non-distilled draft model.The passage attributes this sensitivity to disruption of key-query similarity distributions.Source: §F.5 · p. 10
• MLP_ONLY maintains the unquantized baseline acceptance rate of 1.32 ± 0.04% while reducing active draft memory footprint by 46.1%.The analysis informs a strategy that keeps attention layers at higher precision and applies ternary kernels to dense MLP blocks.Source: §F.5 · p. 10
• The resulting SIMD strategy applies AVX2 ternary kernels exclusively to dense MLP blocks while retaining higher precision in attention layers.This targets the layers identified as resilient to low-bit quantization.Source: §F.5 · p. 10

F.6 Post-Training Quantization Sweep (Layer-Skipped Baseline)

• 1.70x to 2.80x: Figure 4 reports that SubTernary’s physical decoding speedup increases as the target model footprint grows.Source: Figure 4 · p. 11
• Table 13 covers a per-layer quantization sensitivity sweep using a non-distilled, zero-shot baseline, with mean and standard deviation over five runs.Source: Table 13 · p. 11
• Without student-teacher distillation, skipped layers compound representational shifts until the final vocabulary logits are misaligned with the target model.The residual stream carries the shift through successive layers.Source: §F.6 · pp. 10–11
• Table 14 evaluates post-training quantization on a layer-skipped baseline without distillation, reporting the mean and standard deviation over five runs.The sweep examines structural layer truncation and aggressive quantization approaches.Source: §F.6 · p. 10, Table 14 · p. 11